Special stop tokens in stop_sequences doesn't work with GGUFs

LostRuins / koboldcpp

A simple one-file way to run various GGML and GGUF models with KoboldAI's UI

https://github.com/lostruins/koboldcpp

GNU Affero General Public License v3.0

4.36k stars 312 forks source link

Special stop tokens in stop_sequences doesn't work with GGUFs #812

Open noisefloordev opened 2 months ago

noisefloordev commented 2 months ago

I'm trying to use special stop tokens in stop_sequences, but it's not recognizing them as a single token. I'm passing stop_sequence: ["<|im_start|>", "<|im_end|>"] to /api/extra/tokencount, but it doesn't stop on those tokens.

It looks like llama_tokenize is adding the BOS, and changing the third parameter of TokenizeString's llama_tokenize call to add_bos seems to fix this. I'm not sure if that's the right fix, since that parameter has some other effects.

LostRuins commented 2 months ago

There's a planned fix upcoming for this, this should work in the next version.

LostRuins commented 2 months ago

Hi, Should be fixed in the latest version