Open newsletternewsletter opened 1 month ago
If you're seeing the wrong behaviour in llama-tokenize.exe this looks like it's probably an upstream bug?
If you're seeing the wrong behaviour in llama-tokenize.exe this looks like it's probably an upstream bug?
Yes, indeed! I opened a bug ticket there: ggerganov/llama.cpp/issues/8584.
The behavior when the tokenizer adds a space to the first non-special token can be customized via the key tokenizer.ggml.add_space_prefix
.
There are 2 workarounds (https://github.com/ggerganov/llama.cpp/issues/8584#issuecomment-2240032341):
tokenizer.ggml.add_space_prefix=bool:false
.false
.An acceptable workaround: Changing the KV metadata in the GGUF file via a python script works wonders (using a modified gguf-py/scripts/gguf_new_metadata.py
from llama.cpp).
However, trying a KV override via ModelParams.MetadataOverrides
does not seem to work. When adding modelParams.MetadataOverrides.Add(new MetadataOverride("tokenizer.ggml.add_space_prefix", false))
before loading the model via LLamaWeights.LoadFromFileAsync
, the KV override is ignored and the tokenizer adds a space.
This is an upstream bug as it is reproducible with llama-cli
.
It is being fixed upstream: https://github.com/ggerganov/llama.cpp/pull/8614
Gemma 2 2B is released, it's even surpassing GPT-3.5-turbo
However, trying a KV override via
ModelParams.MetadataOverrides
does not seem to work. When addingmodelParams.MetadataOverrides.Add(new MetadataOverride("tokenizer.ggml.add_space_prefix", false))
before loading the model viaLLamaWeights.LoadFromFileAsync
, the KV override is ignored and the tokenizer adds a space.
I tried again with LLamaSharp 0.15.0 and although it has been fixed upstream (https://github.com/ggerganov/llama.cpp/pull/8614), the KV override in LLamaSharp via ModelParams.MetadataOverrides
does not work, neither with models that have tokenizer.ggml.add_space_prefix
set to true
(e.g. Lite-Mistral-150M-v2-Instruct), nor with ones without the key tokenizer.ggml.add_space_prefix
(many old quants).
Description
When tokenizing a text and decoding these tokens, one can see that tokenization now (as of version 0.14.0) adds one additional starting space to
text
for every call ofContext.Tokenize(text, addBos, special)
. This is especially bad if a text is tokenized with more than one call. Version 0.13.0 did not exhibit such behavior. Or at least, it did not add spaces at the start of words, changing their token ids.This seems fine for most models (I saw this when using
trollek/NinjaMouse-2.4B-32L-danube
), but when I use gemma-1.1-2b-it-Q6_K.gguf (from: bartowski/gemma-1.1-2b-it-GGUF) now, it's not working anymore. The prompt was:Validating with
tokenize
from llama.cpp b2985 (used in LlamaSharp Version 0.13.0):Interestingly the token at position 2 with id 2425
' user'
adds a starting space to'user'
(id 1645).But even the latest llama.cpp b3412 does not work correctly, look at token at position 2 with id 968
' <'
:Is there a way to completely prevent extra spaces from being added by tokenization anywhere? I will tokenize them by hand if necessary. 😉
Reproduction Steps
Write the prompt (see above) to
prompts.txt
and run:for llama.cpp b2985:
or for llama.cpp b3412:
Environment & Configuration
Known Workarounds
I would love to know!