How to upload documents and epub

[ACarloGitHub]

Hi, I read that questions about how Memoir works should be placed here. I therefore ask what is the correct procedure for uploading documents to memoir. Through Oobabooga chat? My second question is how to integrate the possibility of loading epub. I noticed that the two features I'm asking about have disappeared from the github home page.

Thanks

[brucepro]

Are you using the main or the development branch? The full rag system is on the development branch. I just about have all the bugs worked out with the rag system, so I will be merging the branches within the week.

[ACarloGitHub]

Thank you for your reply. I'm using the main branch. If you recommend it, I will download the development branch.Is there a particular procedure for uploading documents?

[brucepro]

So in the dev branch there is a full rag implementation that you can use the gui to upload a specific document, or use the [GET_URL] command to import it to rag. Also all the ingest commands. I would recommend you wait until I do the merge as there is a bug I need to find and fix in the rag implementation that adds the last rag look-up data into the longterm memory insertion. I need to be better with var names. Will have that specific bug fixed today in dev branch so once you see that commit, then you can try the dev branch, or wait for the new features. After using the RAG system for a bit, I have needed to adjust a few things. New features are select specific document to only perform rag on, use the entire database or use specific subject domains. Right now when you ingest a document into the RAG system it goes into the main qdrant rag database, so once you hit 16k or so entries, it can be a bit overwhelming. Also the new q/a memory creation directive seems to be providing better memory look-up. So guess short answer is..if it can wait a day or two, check out the dev branch or wait for the new full release with the new features. If you describe your use case I may be able to provide better info too.

[ACarloGitHub]

First of all, thanks for your reply. And thank you also for thinking of a tool like Memoir. My use case is looking for a tool that helps me with creative writing. I would like to upload chapter drafts, character names with their description, locations or some sources to the database and then discuss them with the AI. In a bright future, the ideal would be to upload entire books and be able to assimilate the styles of the authors used as models or inspiration. I write in Italian and use Llama3, with which among other things there are some configuration problems. I also use a modified version of Mistral, Cerberus. This version has a fine tuning with an Italian database. On Ollama he writes in perfect Italian but on Oobabooga he loses a bit of style. Memoir is the best tool, out of the box. I've tried others, but there's always some problem. I have a lot of doubts. I don't know if my dream of having a virtual assistant for creative writing is feasible. I don't know if it's better to train it with RAG or Fine Tuning. I'm experimenting with many ways and, again, Memoir is something that would be perfect in the future. I have a PC with Nvidia 3090 and ryzen 5800. I use Windows 11. I'm not a developer. I'm just an enthusiast. I am very grateful for the work you have done so far.

[brucepro]

Ahh, you will love some of the future features I have planned. Introduction of knowledge graphs that can be used for world knowledge and data. Implementing a progress task list that the a.i. can use to keep track of where it was on a specific task. Also in the memory system I am playing around with some features that help it keep the context blocks such as including the last memory saved in the new memory creation so that it retains the full context. I am also looking at some fun sub-systems in Memoir+ to aid in task completion that uses a type of agent creation system. For your specific use case right now, I would suggest the dev branch and then just upload your .epubs that contain all the knowledge you want it to have. The issue with vector databases is that without a q/a pair related to the content in the vector store it gets hard to find the data. So I am adding in some logic into the memory and the rag look-ups that includes keyword search too.

[ACarloGitHub]

I'm adding another comment, hoping to give you more feedback. As a creative writing assistant, it would be nice for the AI to be able to summarize the various drafts, remember the characters in the book, and any location. I also opened another thread asking if it was possible to have multiple users or topics, with separate databases. In fact, while I was using Memoir to evaluate the drafts with the AI, in a sort of brain storming, it occurred to me that it would be nice to be able to simply chat with her, without necessarily burdening the database with useless information. I also thought about having separate databases for the various chapters, or just for the locations or just for the characters. Plus, after using Memoir, it occurred to me that it would be great to show it to my friends, advertise it. But my book project is personal and it is work that I don't want to share yet. So it would be nice to be able to have more users and databases.

[brucepro]

Thank you, always open to ideas.