search

v2rockets / Loyal-Elephie

Your Trusty Memory-enabled AI Companion - Multilingual RAG chatbot optimized for local LLMs | OpenAI API Compatible
MIT License
208 stars 19 forks source link

*Save* doesn't work after around 40+ messages-long exchange #10

Closed hennas-waifson closed 2 weeks ago

hennas-waifson commented 3 weeks ago

Hello, in the terminal window, I see that, at some point after around 40 ai and user messages, instead of posting a new user message, the previous user message is displayed as input to the model (basically, same message for some reason is being re-sent to the model perpetually after around 40 plus messages). Consequently, not possible to save the session. The workaround is to delete the last several messages to bring the number of messages in the chat down, and then it works properly until this x threshold is reached again.

v2rockets commented 3 weeks ago

yes, there is a default character limit setthings in frontend/pages/api/chat.ts const charLimit = 12000; You can change it directly if your LLM backend can handle larger context length.

hennas-waifson commented 3 weeks ago

TY!

hennas-waifson commented 3 weeks ago

yes, there is a default character limit setthings in frontend/pages/api/chat.ts const charLimit = 12000; You can change it directly if your LLM backend can handle larger context length.

Does this also set the overall max context for the prompt? and where do you have other parameters like how many times to conduct search, how many RAG results to include? TY

v2rockets commented 3 weeks ago

yes, there is a default character limit setthings in frontend/pages/api/chat.ts const charLimit = 12000; You can change it directly if your LLM backend can handle larger context length.

Does this also set the overall max context for the prompt? and where do you have other parameters like how many times to conduct search, how many RAG results to include? TY

Good points! I have planned to expose more parameters in setting files

v2rockets commented 2 weeks ago

some RAG parameters exposed in settings.py

hennas-waifson commented 2 weeks ago

some RAG parameters exposed in settings.py

Hello! I hope all is well! I changed the const charLimit = to a higher number (125000 was the highest I tried), but it begins to send the same user message to the model regardless of what the current message is, be it a regular text or the SAVE command. Please let me know if you want more details.

It would be nice to lift the 1000 char input limit and output limit; it will make the assistant more useful for helping edit longer email drafts, for example. Thank you!