Closed h3ndrik closed 1 year ago
The Token IDs can be referenced against the official llama vocab. For ggml it is embedded directly into the model file itself. I cannot think of an easy way for a user to specify the list of banned tokens though.
Thank you. Okay. the gpt4-x-alpaca i'm testing seems to have that token removed...
Concerning the banned tokens: Would it be okay to introduce another boolean to the generation_inputs struct? next to the stop_sequence? something like suppress_tokens_for_koboldai and just set it with the ArgumentParser?
I could create a pull request, but i wanted to ask first before trying to clutter the code with things that aren't within koboldcpp's intended purpose.
Yeah I'll add a flag for skipping banned tokens. It won't be part of the API, but a launch parameter for koboldcpp itself from the python script
Hi, this has been added in 1.13, as --unbantokens
Well, thank you very, very much.
For other purposes than KoboldAI (e.g. interfacing with the API directly with my python scripts) it would be nice to have something like an argument in the api call, or a cli flag, to enable or disable the suppression of square brackets.
Sure, I can edit the code and recompile... But something easier would be great. I think that would be cleaner code anyways, not to suppress that here, but to sanitize/strip the input in the application that expects it in a certain way.
I'm not sure if a cli flag for koboldcpp or an argument exposed via the API would be the best solution.
https://github.com/LostRuins/koboldcpp/blob/9129e937f92172264ed99065a1ac2a97c1e3a1be/gpttype_adapter.cpp#L515-L519
One more thing, while talking about that code: What happens with the 'end of text' token? I believe that is suppressed, too? Can we have access to it? I want to feed the output into
langchain
and I get useful responses, but the llm keeps filling it's answer with random ramblings until the token limit is reached. And i don't know where the vocabulary is stored. In the ggml file? Because with the model i'm fiddling around, i don't ever get something like<|endoftext|>
or</s>
. Can I overwrite the string representation of token 2 somewhere? And/or add it to the 'stop_sequence'?