Hi! This tool is the best wrapper I've found so far, however it seems to have a critical context issue.
I was having a long conversation, and at one point the thread seemed to be lost—I looked at the network request, and it seems like my first message was included, and my last one, but all the N intermediate messages (probably > 8k) were omitted. This happened before, but at the time I thought it was due a bug with the fact that I used the "edit" feature.
I haven't looked at the code yet, but my guess is there's a bug with context management as context rolls over the max context for a model. I see in the logs "[context trimmer] trimmed -6548 tokens from context". It seems like, after this happens, it doesn't even respond to my new messages—it always responds to my first message, and ignores my latest.
I know context management is hard and has tradeoffs, but perhaps something where the earliest messages are dropped would be better? Progressive summarization with GPT 3.5 is also a good strategy. A section could be removed from the early middle, a summary generated, and replaced with [Your context was trimmed here. Summary of lost context: ...].
It looks like the linked site is dead and there's been no updates in 6 months. Which is totally ok, there's no obligation to maintain an open source tool and I know it's a thankless job :) I might take a stab at fixing this myself later, if I have time. Can you let me know if you are still planning on maintaining this project?
Hi! This tool is the best wrapper I've found so far, however it seems to have a critical context issue.
I was having a long conversation, and at one point the thread seemed to be lost—I looked at the network request, and it seems like my first message was included, and my last one, but all the N intermediate messages (probably > 8k) were omitted. This happened before, but at the time I thought it was due a bug with the fact that I used the "edit" feature.
I haven't looked at the code yet, but my guess is there's a bug with context management as context rolls over the max context for a model. I see in the logs "[context trimmer] trimmed -6548 tokens from context". It seems like, after this happens, it doesn't even respond to my new messages—it always responds to my first message, and ignores my latest.
I know context management is hard and has tradeoffs, but perhaps something where the earliest messages are dropped would be better? Progressive summarization with GPT 3.5 is also a good strategy. A section could be removed from the early middle, a summary generated, and replaced with [Your context was trimmed here. Summary of lost context: ...].
MemGPT seems promising, but a lot more complex.
It looks like the linked site is dead and there's been no updates in 6 months. Which is totally ok, there's no obligation to maintain an open source tool and I know it's a thankless job :) I might take a stab at fixing this myself later, if I have time. Can you let me know if you are still planning on maintaining this project?