llama-server not using GPU

abetlen / llama-cpp-python

Python bindings for llama.cpp

https://llama-cpp-python.readthedocs.io

MIT License

8.16k stars 970 forks source link

llama-server not using GPU #1826

Open RakshitAralimatti opened 1 week ago

RakshitAralimatti commented 1 week ago

After I install llama-cpp-python-server with cuda support and run python3 -m llama_cpp.server --model starcoderbase-3b/starcoderbase-3b.Q4_K_M.gguf --n_gpu_layers 10 The GPU is not getting used its running on the CPU