4096 token limit for chat note

[JustusPan]

I noticed the web version of gpt accept 4096 token each sentence, but one session allow us talking continuously with a quiet long context. but this plugin limit the whole context to 4096 token, is this the limitation of API itself?

[lukemt]

I've thought and read about this a lot, and I'm still not quite sure if there is a solution. One obvious thing one can do is cut off the beginning of the conversation but often times the beginning is the most important part. I have read that ChatGPT has some kind of memory vectors that it keeps during the conversation so it can remember things outside of the context window. But I believe there is no way to achieve this with the ChatGPT API yet. If anyone knows more please let us know

[bramses]

hi @JustusPan, yep exactly as @lukemt says there is no one agreed upon solution, and the API hard caps at 4096

Some things you might try:

1. sliding window: highlight the last n messages and use ChatGPTMD to start a new chat from highlighted text.
2. Ask ChatGPT to summarize the messages (like the Infer Title command) and create a new chat and put the summary as a system command.
3. Reword your questions/typing so that the source material is shorter
4. Use GPT 4 (twice the tokens lol!)

These are all subjective/user dependent, and ChatGPT MD as of now any addition like that would only make the plugin more confusing, imo. But one day, if there is a (mostly) agreed upon solution on how to deal with token limits, I'll add it!

[hayoial]

In the present moment is it possible to add an option for maximum historical context, such as last 0-20 previous messages, so that not all the context (messages) need to be sent every time? This way, we can keep the conversation going in the same window.

[bramses]

hi @hayoial, i can look into it as potentially some sort of setting. can you open an issue with your suggestion?