Was wondering if there's a way to pass --ignore-eos flag to koboldcpp

LostRuins / koboldcpp

Run GGUF models easily with a KoboldAI UI. One File. Zero Install.

https://github.com/lostruins/koboldcpp

GNU Affero General Public License v3.0

4.66k stars 334 forks source link

Was wondering if there's a way to pass --ignore-eos flag to koboldcpp #149

Closed dogjamboree closed 1 year ago

dogjamboree commented 1 year ago

I've found that with certain models, especially OpenAssistant ones, generation seems to get cut off prematurely if I don't use the --ignore-eos flag, or at least this is my perception. Answers seem to be much longer and more complete and answers aren't cut off mid-sentence as often.

Is there a possibility of implementing this flag is koboldcpp? Thanks!

LostRuins commented 1 year ago

You have the --unbantokens flag which is the opposite. When this flag is not set, the EOS token is ignored. If your answers are being cut off be sure it's not because a stopping sequence has been triggered, multiline chat restrictions, or the gen amount is too low.

dogjamboree commented 1 year ago

Ahh, good to know. I'm sure you're right, it happens randomly with different models so I wasn't sure, I just wanted to check.