search

trzy / llava-cpp-server

LLaVA server (llama.cpp).
MIT License
173 stars 9 forks source link

GPU Support #1

Closed gusanmaz closed 11 months ago

gusanmaz commented 11 months ago

I am able to run llava-cpp-server on CPU without an issue. Is there a way to run llava-cpp-server on GPU?

trzy commented 11 months ago

I believe llama.cpp has some CUDA support. I include llama.cpp as a sub repo so I would recommend looking at their docs to see whether this is true and if so, whether any build flags need to be enabled.

ygean commented 10 months ago

I am able to run llava-cpp-server on CPU without an issue. Is there a way to run llava-cpp-server on GPU?

Hi gusanmaz, what speed can you run on cpu?