retrieval augmented generation (RAG) proof of concept

[psugihara]

I want to be able to do RAG with a set of local files or even my whole filesystem. As a proof of concept, let's try giving FreeChat a long term memory by doing context retrieval from all past messages.

• [ ] embed/index messages in some kind of vector db
• [ ] when a user does a query, search for relevant messages and put them in the system prompt or append to the user's message

The whole thing should of course have no installation steps for users.

[blizzn]

I have similar interests. So you want to RAG the filesystem and which would also mean a folder for books for example. How deep should that inforamtion go, should that LLM be able to tell you what are the most recent books you added and what their content is? give code examples and be able to adjust them to your needs?

[psugihara]

Yes I think all of that would be great! Metadata + content of the files including date added, directory location, etc. I think we'll need to experiment with what's actually useful and prompt format. In my mind this is a key promise of local AI... we learn on all of our private data and the AI "knows" us and what we're working on.

Some examples of where this might be useful

• Asking to write something in our style or a writer's style
• Asking questions about a topic I have collected a lot of documents about
• Asking something about a conversation we've had in the past
• Asking for help in a codebase

Is this what you mean by code examples? Like that you could point it at a codebase and it would give you examples? If so, I like the idea.

Again I think an easy-ish dataset to try first is just the conversation histories (Message objects stored in CoreData/sqlite). That would be easy to grab, require no extra permissions, and allow us to prototype the architecture.

[blizzn]

i think that writers style is more finetuning if its personal. so longterm data. tI m playing around with gpt4 and askyourpdf. they have a new feature knowledge base in which you can put several books and ask questions about it. my gpt4 plugin for askyourpdf doesnt work for the knowledge base but its internal LLM just fine. I m playing around with it right now to think what functionality would be most useful to me. but what i m already missing is that gp4 cant run programs on my machine. i want to tell it, write a program with the knowledge of these books and execute it when, how often. I m not sure about the depth of 7b, 13b models, that I can run mostly, but i like what i saw at askyourpdf. the least functionality i would require of a meChat would be: work with local files, text, json, sql like u said. plus kinda api functionality to other programs i can add that do things when i tell my meChat when to do it. so to automate most task of my daily life.

[cleesmith]

After trying all of the apps out there for AI, mostly for helping with creative writing, FreeChat is appealing to have something local and somehow use my own files along with all that knowledge the AI already has about writing. But for writing, the only consistent "ok" results I have had are with GPT-4 and Claude, and that takes a lot of prompting/editing, FreeChat is way behind those. But I'm definitely interested in: "Asking to write something in our style or a writer's style" ... I have lots of writing as text files (and PDFs, DOCX, Google Docs, EPUBs). I'm not sure it's helpful but there is a website with free old public domain books in several formats: https://standardebooks.org/ebooks. Well, I'm very willing to test things out.

[shavit]

Maybe for the second task, it can use a combination of the last N messages and NER to find relevant information in conversations.