search

LostRuins / koboldcpp

Run GGUF models easily with a KoboldAI UI. One File. Zero Install.
https://github.com/lostruins/koboldcpp
GNU Affero General Public License v3.0
5.3k stars 362 forks source link

EOS token triggered! #1177

Open HarryK4673 opened 1 month ago

HarryK4673 commented 1 month ago

Hi everyone!

I met a problem when generating texts. I set the flags to --contextsize 98304 and when I test it, I input a text with about 30k tokens and set the maximum output tokens to 600. EOS token triggered when it output 8/400 tokens.

(EOS token triggered! ID:128009)
CtxLimit:30989/98304, Amt:8/600, Init:0.12s, Process:268.39s (8.7ms/T = 115.43T/s), Generate:10.71s (1338.2ms/T = 0.75T/s), Total:279.10s (0.03T/s)

Is there any ways to modify the EOS token?

I met this problem only on Google Colab. It's fine on my local computer.

LostRuins commented 1 month ago

EOS means the model wishes to stop generating the current response as it believes it is complete

To override this, set EOS token ban to "Ban"

image

HarryK4673 commented 4 weeks ago

EOS means the model wishes to stop generating the current response as it believes it is complete

To override this, set EOS token ban to "Ban"

image

Thanks for that, but I'm using Google Colab (using the Linux version and command to execute). I did not find any flag related to EOS when ./koboldcpp

LostRuins commented 4 weeks ago

EOS banning is set over the API by the frontend. Are you using the KoboldAI Lite frontend?