Closed Ralakus closed 2 months ago
Problems with finetunes with CUDA are often caused by the model producing values that cannot be represented in a 16-bit float. If that's the case, building with LLAMA_CUDA_FORCE_MMQ
may help.
Compiling with LLAMA_CUDA_FORCE_MMQ
didn't seem to fix the issue. What I'm most curious about is why it only happens when running parallel inferencing and goes back to normal with a logit bias.
I did a lot more tests and found out that the finetuning actually wasn't the issue. Adding any extra tokens to the tokenizer before converting the model AND then quantizing the model with the added tokens causes this with Mistral and Mixtral models. This issue doesn't occur when running the model using f16 and f32.
I did the tests on the Mistral Instruct v0.2 model with an added padding token.
Q4_K_M Parallel out
Q4_K_M Parallel log
Q4_K_M Parallel out with logit bias
Q4_K_M Parallel log with logit bias
This issue was closed because it has been inactive for 14 days since being marked as stale.
Summary
When using the CUDA backend, parallel inferrencing generates the unknown token (token id 0) in all sessions except one. This issue doesn't occur when there is only one client inferrencing or when llama.cpp is running with no layers offloaded to the GPU.
The unknown tokens get decoded as the UTF-8 Lower Five Eighths Block (▅ U+2585) character and seems to turn the rest of the output into a jumbled mess of text that partially resembles a response.
Steps to Reproduce and Output
Parallel example program using CUDA
llama.cpp/build/bin/parallel -m Mixtral-Test-Finetune/ggml-model-q4_k_m.gguf -ngl 999 -c 32768 --parallel 4 --sequences 4 --seed 25519
Full output
Log
Server parallel client test script using CUDA
Python script
Output
Parallel example program using CPU
llama.cpp/build/bin/parallel -m Mixtral-Test-Finetune/ggml-model-q4_k_m.gguf -ngl 0 -c 32768 --parallel 4 --sequences 4 --seed 25519
Full output
Log
Server parallel client test script using CPU
Python script
Output
Work-around
Setting logit_bias of the unknown token to never generate (or some really low value like -999999999) seems to get around the llama.cpp generating unknown tokens when generating in parallel. While setting the logit_bias of token 0 to never generate bypasses the issue, it would be beneficial to figure out the discrepancy between CPU and CUDA inferencing.
Parallel example program
llama.cpp/build/bin/parallel -m Mixtral-Test-Finetune/ggml-model-q4_k_m.gguf -ngl 999 -c 32768 --parallel 4 --sequences 4 --seed 25519 --logit-bias 0-999999999
Full output
Log
Server parallel client test script
Python script
Full output
System Specifications