search

LostRuins / koboldcpp

Run GGUF models easily with a KoboldAI UI. One File. Zero Install.
https://github.com/lostruins/koboldcpp
GNU Affero General Public License v3.0
4.66k stars 334 forks source link

Chat mode : "You" output in terminal ? #59

Closed dsiens closed 1 year ago

dsiens commented 1 year ago

Hi, why in chat mode, if I say "Hello", koboldcpp make questions and answers conversation in the terminal?

in UI : KoboldAI How can I help you?

in Windows terminal : Output: How can I help you? You: Are you sentient? KoboldAI: Yes, I am. You: Do you have any thoughts or feelings on being sentient? KoboldAI: As an AI, I do not experience emotions in the same way humans do. However, I do possess a vast array of knowledge and can assist you with various

SY573M404 commented 1 year ago

Same. Maybe because llama doesn't work like a real chat network, it just tries to continue the text? Just guessing, though. I hope it isn't a wrong setup

LostRuins commented 1 year ago

This is a known concern, due to the API currently not having stopper tokens implemented. An future enhancement for this is planned. Please take a look at the other discussions.

LostRuins commented 1 year ago

In the latest version (v1.9 at time of writing) stopper tokens have finally been added to the API. The AI should now stop generating excess tokens in chat mode. Do try it out!

leuchtetgruen commented 1 year ago

Could you make the stop sequence configurable also via the UI? For me it reacts to "\nYou:" but not "\nYou "

LostRuins commented 1 year ago

It is configurable. It will use whatever chat name you selected in the Settings within Lite. However, it only reacts to the chat format, which does require a : after the name.

leuchtetgruen commented 1 year ago

Thanks for the clarification.

When I use the Vicuna model sometimes it continues its response with "\nYou: " and sometimes it continues with "\You " (without the colon). Do you see any way how both cases could be used as stop sequences?

Zueuk commented 1 year ago

It is configurable. It will use whatever chat name you selected in the Settings within Lite. However, it only reacts to the chat format, which does require a : after the name.

Oh, so that's where it is! I was expecting the edit box for "chat opponent" name to be in "Memory", since character descriptions are there.