Organize Full Technology Stack

[TanGentleman]

• Base repository is forked from: https://github.com/get-convex/turbo-expo-nextjs-clerk-convex-monorepo/
• Web UI is written in typescript, and is React native.
• Backend database is powered by Convex and allows for real-time and transactional behavior (important at scale)
• LLM and Embedding components are provider-agnostic, but most development is done using TogetherAI for compute
• For MVP, Vectara simplifies document ingestion/chunking/retrieval with RAG-as-a-service and a compatible SDK.
• Moving forward, Convex Auth can replace Clerk for authentication!

Orchestration Concerns:

• There have been some great improvements in recent LLMs, and function-calling is now baked natively into models like the Llama-3.1 family. For instance: https://docs.together.ai/docs/llama-3-function-calling is a great resource by TogetherAI for handling such a test using a specific subset of models. This, however, is not very flexible to LLM or provider! I should not invest too many hours in non-robust components, but for MVP/proof-of-concepts, they work.
• All steps should conform to LangGraph framework in any given workflow. This means approaching each task with state organized and the solution space well defined. This will be more important as RAG gets more complicated, like adding filter kwargs to a search in a vector space.

[TanGentleman]

The flow will look a little something like this:

1. User visits textbook-ai website, signs in, and is directed to app page
2. User uploads a PDF (Like Intro to Physics) into memory
3. A TextbookSchema is generated and assured to be valid (Uses LLM component)
4. Chunk the textbook with appropriate metadata (page number, other relevant fields) - Note that a function getMetadataByPage would still need information that should be collected @ step 3. Something like mapping the page ranges to the chapter/sections (Should be in table of contents)
5. Add these chunks and their metadata to a Convex table
6. Create embeddings for each chunk

At this point, indexing is done!

[TanGentleman]

After indexing a textbook, here's an abstracted version of what flows should look like: INPUT: The active settings, the current state, and the user input (button press or string input) OUTPUT: GOAL (Retrieved content, Content + Text, or just Text (covers fail case)

The flow:

1. Collect inputs
2. Identify GOAL, or fail (if input can't be resolved to a valid goal)
3. Begin appropriate LangGraph workflow!
4. Stream steps to completion or fail (Ensures deterministic failure behavior with handled side effects)
5. Serve GOAL (e.g. Excerpts of the textbook content, with the appropriate pages/sections provided, + add a summary cuz why not, AI is gimmicky but it can be helpful)

The reason to make this a touch more complicated with LangGraph (at least for now) is because of the side effects when handling eval functions or other substeps that may fail, but can be soundly reasoned through.

[TanGentleman]

I still agree with most of the design patterns here! Here's a lil AI generated strategy based on these notes:

Here's a comprehensive product plan organized from your technical notes:

TextbookAI - Product Development Plan

Product Overview

TextbookAI is an intelligent learning platform that transforms standard textbooks into interactive, searchable content with AI-powered features. The platform allows users to upload educational materials and interact with them in novel ways through advanced AI capabilities.

MVP Feature Set

Core Functionalities

1. User Authentication

   • Secure sign-in system
   • Personal user dashboard

2. Document Processing

   • PDF upload capability
   • Automatic textbook structure analysis
   • Smart content indexing

3. Interactive Features

   • Intelligent content search
   • Context-aware responses
   • Real-time content navigation

Technical Architecture (MVP)

Frontend

• React Native (TypeScript)
• Responsive web interface
• Real-time updates support

Backend

• Convex database for real-time data handling
• TogetherAI for AI compute
• Vectara for document processing (RAG-as-a-service)
• Clerk for authentication (Phase 1)

User Flow

1. Onboarding

   • User visits website
   • Creates account/signs in
   • Arrives at dashboard

2. Content Upload

   • User uploads PDF textbook
   • System processes document
   • Automatic content structuring

3. Interaction

   • Search within textbook
   • Receive AI-enhanced responses
   • Access relevant excerpts with page references

Development Phases

Phase 1 (MVP)

• Basic authentication
• PDF upload and processing
• Simple search and retrieval
• Basic AI interactions

Phase 2

• Enhanced AI capabilities
• Advanced search features
• Improved content structuring
• Migration to Convex Auth

Phase 3

• Additional document format support
• Advanced analytics
• Collaborative features
• Custom AI model optimization

Technical Considerations

Scalability

• Real-time database capabilities
• Transactional behavior support
• Efficient content processing

Security

• Secure authentication
• Data encryption
• Privacy compliance

AI Integration

• Provider-agnostic architecture
• Flexible LLM integration
• Structured workflow management

Success Metrics

1. User adoption rate
2. Document processing speed
3. Search accuracy
4. User engagement time
5. System reliability

Timeline

• MVP Development: 2-3 months
• Testing & Refinement: 1 month
• Initial Launch: End of Q1

Resource Requirements

1. Development Team
2. Cloud Infrastructure
3. AI Computing Resources
4. Testing Environment

This plan provides a structured approach to building TextbookAI, focusing on delivering core functionality in the MVP while maintaining flexibility for future enhancements. The architecture is designed to be scalable and maintainable, with clear progression paths for feature development.

My thoughts: The timeline/resource requirements will certainly be affected by scale, but for now, we're not far from the complete vision, even abstracted past textbooks! I found it a surprisingly decent layout for a potential strategy :)