Organize Full Technology Stack

[TanGentleman]

• Base repository is forked from: https://github.com/get-convex/turbo-expo-nextjs-clerk-convex-monorepo/
• Web UI is written in typescript, and is React native.
• Backend database is powered by Convex and allows for real-time and transactional behavior (important at scale)
• LLM and Embedding components are provider-agnostic, but most development is done using TogetherAI for compute
• For MVP, Vectara simplifies document ingestion/chunking/retrieval with RAG-as-a-service and a compatible SDK.
• Moving forward, Convex Auth can replace Clerk for authentication!

Orchestration Concerns:

• There have been some great improvements in recent LLMs, and function-calling is now baked natively into models like the Llama-3.1 family. For instance: https://docs.together.ai/docs/llama-3-function-calling is a great resource by TogetherAI for handling such a test using a specific subset of models. This, however, is not very flexible to LLM or provider! I should not invest too many hours in non-robust components, but for MVP/proof-of-concepts, they work.
• All steps should conform to LangGraph framework in any given workflow. This means approaching each task with state organized and the solution space well defined. This will be more important as RAG gets more complicated, like adding filter kwargs to a search in a vector space.

[TanGentleman]

The flow will look a little something like this:

1. User visits textbook-ai website, signs in, and is directed to app page
2. User uploads a PDF (Like Intro to Physics) into memory
3. A TextbookSchema is generated and assured to be valid (Uses LLM component)
4. Chunk the textbook with appropriate metadata (page number, other relevant fields) - Note that a function getMetadataByPage would still need information that should be collected @ step 3. Something like mapping the page ranges to the chapter/sections (Should be in table of contents)
5. Add these chunks and their metadata to a Convex table
6. Create embeddings for each chunk

At this point, indexing is done!

[TanGentleman]

After indexing a textbook, here's an abstracted version of what flows should look like: INPUT: The active settings, the current state, and the user input (button press or string input) OUTPUT: GOAL (Retrieved content, Content + Text, or just Text (covers fail case)

The flow:

1. Collect inputs
2. Identify GOAL, or fail (if input can't be resolved to a valid goal)
3. Begin appropriate LangGraph workflow!
4. Stream steps to completion or fail (Ensures deterministic failure behavior with handled side effects)
5. Serve GOAL (e.g. Excerpts of the textbook content, with the appropriate pages/sections provided, + add a summary cuz why not, AI is gimmicky but it can be helpful)

The reason to make this a touch more complicated with LangGraph (at least for now) is because of the side effects when handling eval functions or other substeps that may fail, but can be soundly reasoned through.