Closed davideuler closed 6 months ago
You need to build it with LLAMA_METAL=1, and also you need to specify GPU layers to offload. did you do that?
You need to build it with LLAMA_METAL=1, and also you need to specify GPU layers to offload. did you do that?
Thanks, I rebuilt koboldcpp with LLAMA_META=1, and start the service with GPU layers. Now it works on Meta GPUs. It's super fast now.
python3.10 /Users/david/workspace/koboldcpp/koboldcpp.py --gpulayers 80 --threads 8 miqu-1-70b.q5_K_M.gguf 8501
Run by koboldcpp
When I run koboldcpp for miqu-70b gguf, It is super slow, about 1 token/s. The command to run is as:
Even if I specify parameter for gpu layers, it is the same.
And I check the GPU usage by
sudo asitop
, then found that the GPU power usage is almost near to 0. koboldcpp is using CPU for inference on my Mac m1. With a very long wait time(about 30 seconds), and slow inference speed. (about 1 tokens/s)Run by llama.cpp
While when I run it by llama.cpp, it is fast with little wait time. About 2-3 seconds wait time. The inference speed is near 5 tokens/s. And most of the power usage is spent on the GPUs.
The output :