search

LostRuins / koboldcpp

Run GGUF models easily with a KoboldAI UI. One File. Zero Install.
https://github.com/lostruins/koboldcpp
GNU Affero General Public License v3.0
4.79k stars 340 forks source link

Problems on Android-GPU #974

Open Martmists-GH opened 2 months ago

Martmists-GH commented 2 months ago

I'm struggling getting GPU to work on Android. Device is OnePlus 8T, and I'm keeping the app in the foreground (to ensure it's not getting killed).

Using --useclblast 0 0 on a small model results in ggml_opencl: clGetPlatformIDs(...) error -1001 at ggml-opencl.cpp:989 before it can launch the backend.

Using --usevulkan 0 on a small model results in the model seemingly getting loaded and the backend loading, but making any request results in terminating due to uncaught exception of type vk::DeviceLostError: vk::Queue::submit: ErrorDeviceLost

OpenCL seems to work fine, though very slow to process any prompt.

gustrd commented 2 months ago

Sadly there is no stable way to use Android GPUs at this project or the original llama.cpp.

You can check the historical issues here or there.

LostRuins commented 2 months ago

Yeah, your best bet is just using the default cpu mode in termux. It's still fast enough to run 3b at decent speeds and 7b at a slow speed.