Open generic-placeholder-name opened 1 month ago
I just stumpled upon this problem when using a Qwen2 based model myself, but it's a GGUF model in my case. There are open issues in the llama.cpp
repo about it.
There are two things important here it seems:
-flash_attn=True
)-DGGML_CUDA_FORCE_CUBLAS=on
when installing llama-cpp-python
)Maybe this will help you fix the issue.
Prerequisites
Please answer the following questions for yourself before submitting an issue.
Expected Behavior
Model generates normal output when using GPU acceleration.
Current Behavior
Model instead generates gibberish. Moreover, when using a grammar, the model encounters an assertion error.
For example, invoking the following code fragment:
With no GPU acceleration, the model produces the following output:
With GPU acceleration, the model produces the following output:
Additionally, when using a string grammar, I encountered this assertion error:
Environment and Context
My laptop has a NVIDIA GeForce GTX 4060 GPU and is running with CUDA 12.5.1. OS: Windows 11 Home version 10.0.22621 build 22621.
Python version:
Make version:
GCC version:
Failure Information (for bugs)
Please help provide information about the failure if this is a bug. If it is not a bug, please remove the rest of this template.
Steps to Reproduce
llama-cpp-python
with CUDA.Failure Logs
There are no failure logs - the program just returns nonsensical output.