Allow for infinite/compressed chat history

[krassowski]

Problem

Previously the chat kept only two messages and that was hard-coded; with https://github.com/jupyterlab/jupyter-ai/pull/943 in we now have a AiExtension.default_max_chat_history setting which is great as it allows to increase the number from two to say 10. However context for a longer history will still be lost altogether. It is impossible to set memory to infinite, even if the model caches tokens.

Proposed Solution

• Implement memory compression (e.g. batches of messages exceeding max_chat_history get fed into the LLM asking it to compress say each 10 messages into a single summary message), or
• Allow to set AiExtension.default_max_chat_history (if the above solution is judged to be too complex, then the custom providers of models would be able to implement this on their side, provided that they receive all past messages)

Additional context

Langchain has a dedicated example on how to implement summarization for chat history here:

https://python.langchain.com/v0.2/docs/how_to/chatbots_memory/#summary-memory

[krassowski]

To spell it out, default_max_chat_history cannot be set to infinite as of today because it is defined as an Integer (and math.inf is a float) and even if it was a Float it would later fail on BoundedChatHistory expecting an int. I think the solution here could be to treat None as a special value. Thoughts?

[dlqqq]

I think the solution here could be to treat None as a special value. Thoughts?

That makes sense to me. 👍

[srdas]

1. Setting to None is a good idea.
2. One thing to keep in mind is that with very long chat memory, you may exceed the LLM's input context window, in which case the memory will be truncated, keeping the earliest chat exchanges, not the most recent ones, so it will need to be handled if we want the most recent ones (without the summarization idea).
3. For the idea to summarize chunks of the history rather than carrying all of it (to reduce number of tokens and save costs), we need to extend the default setting instructions at startup (jupyter lab --AiExtension.default_max_chat_history=2) to also include a parameter for size of memory trail to summarize (or is everything beyind the default_max_chat_history?).

[krassowski]

For the idea to summarize chunks of the history rather than carrying all of it [...] we need to extend the default setting instructions at startup [...] to also include a parameter for size of memory trail to summarize

Yes, that was my thinking too. Because there are couple of ways to implement compression, e.g.:

• ask LLM to summarize every 10 messages into one and append the new one to a history buffer; once that buffer gets 5 messages then ask it to summarize it into a single message, or
• we could make this happen on a time-based interval, e.g. every 5 minutes if there are more than 12 messages summarize them into one, or
• we could have only one "summary" message and periodically ask the model to update it

And each of these would have different set of parameters. I do not want to put too much compression logic into jupyter-ai to avoid making it hard to maintain, maybe let's have some simple default and allow swapping it out for something more advanced in extensions?