search

LostRuins / koboldcpp

Run GGUF models easily with a KoboldAI UI. One File. Zero Install.
https://github.com/lostruins/koboldcpp
GNU Affero General Public License v3.0
4.66k stars 334 forks source link

koboldcpp not using GPU #723

Closed Fuckingnameless closed 5 months ago

Fuckingnameless commented 5 months ago

when i run the latest prebuilt binary with: ./koboldcpp-linux-x64 --usecublas

it uses the cpu even though i have the latest drivers/cuda, am i missing some flag or config?

LostRuins commented 5 months ago

Did you set --gpulayers? If yes, share the terminal output when the model is loaded

Fuckingnameless commented 5 months ago

nope, i didn't set cause i wanted to use gpu only

LostRuins commented 5 months ago

If you want to get a generation speedup, you should offload layers to GPU.

Fuckingnameless commented 5 months ago

If you want to get a generation speedup, you should offload layers to GPU.

i thought i'd read somewhere that if i don't specifiy it it should go all to the gpu but i was mistaken, thanks for the heads up, now trying to get a bit more performance on this old card