Closed MathiasGrund closed 11 months ago
the binding here is much more on par with upstream compared with the python one. Can you tell if older version works and this is just a regression?
A better data point to pinpoint the issue would be if you could try with llama.cpp directly rather then the python binding. Thanks!
The no kernel image is available for execution on the device
is a local issue for my setup (now fixed!), the feedback here is just that despite running the example successfully, the thing actually did not work. I would suggest adding -ngl 1
to the example codes to force any cuda runtime issue to be exposed immediately, at least that would have made it fast for me to root cause it.
Running the cuBLAS example fails with the following error if you actually use the GPU:
The model works fine with llama-cpp-python so should not be the culprit. It works fine without
-ngl
but then it doesn't use GPU acceleration.