Closed SeanXu1219 closed 1 year ago
As we all know, chatGPT3.5 has 4K tokens, while GPT4 has 8K/32K. However, currently, I have implemented my own chatAI based on API and embedding, but its performance in multi-turn conversations is not satisfactory. The TRIME memory enhancement method has inspired me, and I wonder if it can replace the recent N-turn conversation + embedding (which is heavily limited by token length) to enhance memory and improve the performance of private domain chatAI in multi-turn conversations.
@SeanXu1219 For long-context modeling, I think you should look at the long-term memory variant of our approach called TRIME_long. I think it is a promising solution to incorporate long-range context with proper training, although our paper only shows perplexity so far.
Hi, thanks for your interest! But sorry I may not fully understand your question. It'd be helpful if you could elaborate on what "improve understanding of context" means here!