Closed dr11z3r closed 1 year ago
I think [ and ] are special characters for koboldcpp which will never be generated. They're reserved for encapsulating things like the Author's Note.
I just checked the source and could not find any special usage of [ and ]. Maybe the issue might be related to the sampling (maybe the sampling order?) Interestingly, if I include [ in the prompt, it generates the ].
@horenbergerb is correct. KoboldAI and KoboldCpp set the logit value of the Opening Square Bracket [
token to zero intentionally, this is done because KoboldAI uses the square bracket to encapsulate author note and word information.
If you are building from source and wish to disable this behavior, you may modify lines 226 of llama_adapter.cpp
After commenting out that line and recompiling from source, it seems to work fine now.
Even though I am using the same sampling parameters as in the llama.cpp repo, the generation output in koboldcpp is significantly worse. It feels like koboldcpp is ignoring the prompt format.
llama.cpp repo
Command & Env:
prompt.txt:
Some outputs:
The output always starts with [(...)] every time I test it on the llama.cpp repo, but it never does with koboldcpp. It feels 'strange', like there is a negative bias towards '[' and ']' tokens. Here are some koboldcpp outputs using same temp, top_k, top_p and rep penalty, attempted adjusting the rep penalty context length but was not effective:
Env
Model: ggml-vicuna-13b-1.1-q4_0.bin Windows 10 Ryzen 5 5600X + 3060 TI + 64 GB RAM