Closed dogjamboree closed 1 year ago
You have the --unbantokens flag which is the opposite. When this flag is not set, the EOS token is ignored. If your answers are being cut off be sure it's not because a stopping sequence has been triggered, multiline chat restrictions, or the gen amount is too low.
Ahh, good to know. I'm sure you're right, it happens randomly with different models so I wasn't sure, I just wanted to check.
I've found that with certain models, especially OpenAssistant ones, generation seems to get cut off prematurely if I don't use the --ignore-eos flag, or at least this is my perception. Answers seem to be much longer and more complete and answers aren't cut off mid-sentence as often.
Is there a possibility of implementing this flag is koboldcpp? Thanks!