search

LostRuins / koboldcpp

A simple one-file way to run various GGML and GGUF models with KoboldAI's UI
https://github.com/lostruins/koboldcpp
GNU Affero General Public License v3.0
4.36k stars 312 forks source link

Need help with stop sequences #790

Open inspir3dArt opened 2 months ago

inspir3dArt commented 2 months ago

I try to make the LLM stop it's reply when it is about to write the following things:

(End) [END] Note: (Note: *** {{user}}: You: User: I tried to list them using --bantokens, but it says in the terminal "<" is unexpected. I tried it using the set grammar option in the settings, tried to separate them with a comma, put every word in quotes, and so on, but it doesn't work. I know I can edit the last reply in chat, but it often causes that a lot of tokens get processed again, and that takes quite some time. How can I make the LLM stop at this outputs I'm chat mode, because what it writes from there on is usually Off-Topic or out of character and ruin's the roleplay. Edit: The post don't show words like START and STOP in between "<" and ">" for some reason, I want the LLM to stop at that outputs too
LostRuins commented 2 months ago

You should use the "Extra Stop Sequences" in the UI image Set it to this string (End)||$||[END]||$||Note||$||***||$||You:||$||User:

inspir3dArt commented 2 months ago

How do I find that option? I have installed koboldcpp in termux on Android.

Screenshot_2024-04-19-08-50-56-17_82cebf612ef3f9a8762672c971adaed4

LostRuins commented 2 months ago

Click "Memory" or "Context" button outside

inspir3dArt commented 2 months ago

Works perfectly from my first test, many thanks for your help. It would be nice if this could be saved in the future in the browser catch, like the settings.

I just realized that I used the wrong place to ask my question, since it is not a bug report, sorry for that, I'm not used much to GitHub. I will use the discussion section for this kind of questions in the future.