search

enricoros / big-AGI

Generative AI suite powered by state-of-the-art models and providing advanced AI/AGI functions. It features AI personas, AGI functions, multi-model chats, text-to-image, voice, response streaming, code highlighting and execution, PDF import, presets for developers, much more. Deploy on-prem or in the cloud.
https://big-agi.com
MIT License
5.32k stars 1.2k forks source link

[Context Window] A method to break through the limit of tokens per conversation. #94

Open PaRaD1SE98 opened 1 year ago

PaRaD1SE98 commented 1 year ago

Let gpt summarize and shorten the current conversation after every few questions and answers, so that we don't have to manually choose and delete the contents frequently.

Make it a settings switch option so we still have the precise control version.

enricoros commented 1 year ago

Can have a menu to 'squash' the conversation too.

enricoros commented 1 year ago

How important is this issue to you?

What do you think would be a good ux flow to have this feature without cluttering the UI?

PaRaD1SE98 commented 1 year ago

Not that important, just a bit annoying when having a long chat, because don't want to start over as well as selecting which one to delete.

For the UX flow, currently it behaves like this: image I think we can still give this token length warning, but then start a 'KeyMemory' mode automatically. It should be similar to the autoTitle feature, but allow it to be somewhat longer.

Summarize the current chat history in a limited token length which can be adjusted in the settings modal. (Because a shorter limit -> worse memory quality, larger limit -> shorter future available tokens) I haven't dived into the store-chats.ts, but a rough idea is to control the whole chat history displayed on the UI in a new place of the store. Then we deal with each short chat history with OpenAI API internally.

The delete message feature is still useful for users to manually delete meaningless content (before triggering an auto-summary).

Overall, the chat thread could be a long non-editable history concatenated with a latest editable short history. The first message of latest editable short history is the auto-summary result of the last editable short history. (Treat the summary as a normal 'user' message but allow toggling display and editing like a ‘system’ message)

(Maybe make the non-editable history a different bgcolor) (Or still make all history editable for 'export conversation' feature but give a note that editing old history has no effect on auto-summary result)

enricoros commented 1 year ago

I like your description/idea of different resolutions for the messages. Meaning to keep the originals and the summary. There are a few things that need to figure out (the summarization operation will be slow, and affecting UX), but in theory history can be managed much better.

enricoros commented 10 months ago

Putting also this on the roadmap, because like #216 but not exactly.