search

mit-han-lab / TinyChatEngine

TinyChatEngine: On-Device LLM Inference Library
https://mit-han-lab.github.io/TinyChatEngine/
MIT License
624 stars 58 forks source link

Unable to maintain chat history and continuous chat #66

Open Rkyzzy opened 9 months ago

Rkyzzy commented 9 months ago

Hi! Thanks very much about the great work. Following the instruction provided, I successfully deploy llama7b chat and llama13b chat on my Jetson AGX Orin Device. For single conversation, it works just fine. However, it seems that it does not have the ability to maintain chat history and do continuous chat. It will forget the previous context. Is it the problem with llama 2 model itself or with the engine. Any suggestion on how to solve this? @RaymondWang0 , Thanks in advance!

RaymondWang0 commented 9 months ago

Hi @Rkyzzy, thank you for your interest in our work. The issue you noticed arose due to our previous implementation. But the good news is, we've addressed it. TinyChatEngine should be able to remember past contexts properly now. Please feel free to try it out!