search

ggerganov / llama.cpp

LLM inference in C/C++
MIT License
63.84k stars 9.15k forks source link

Manual CLBlast device and platform number #1368

Closed daniandtheweb closed 1 year ago

daniandtheweb commented 1 year ago

Using CLBlast on some devices (like my Radeon 5700XT) the program defaults to device id 0 and platform id 0, that causes the following error:

Initializing CLBlast (First Run)...
Attempting to use: Platform=0, Device=0 (If invalid, program will crash)
Using Platform: Clover Device: AMD Radeon RX 5700 XT (navi10, LLVM 15.0.7, DRM 3.52, 6.3.1-zen1-1-zen)
fatal error: cannot open file '/usr/share/clc/gfx1010-amdgcn-mesa-mesa3d.bc': No such file or directory

When I manually edit the platform id inside the source to 1 however makes the program behave as expected using the GPU for the prompt evaluation:

Initializing CLBlast (First Run)...
Attempting to use: Platform=1, Device=0 (If invalid, program will crash)
Using Platform: AMD Accelerated Parallel Processing Device: gfx1010:xnack-

My suggestion is to add a launch option to manually choose the desired platform and device id similarly to the --useclblast option of koboldcpp.

sw commented 1 year ago

It should be possible to set the CLBLAST_PLATFORM environment variable to achieve this, but I haven't tested it.

SlyEcho commented 1 year ago

There are two environment variables you can set: GGML_CLBLAST_PLATFORM and GGML_CLBLAST_DEVICE (just numbers)

(Personally I think there should be a command line argument, and it's not really CLBlast that it's controlling, but OpenCL in general. CUDA doesn't have any way to control the GPU selection right now, @slaren?)

slaren commented 1 year ago

CUDA doesn't have any way to control the GPU selection right now, @slaren?

The environment variable CUDA_VISIBLE_DEVICES should work.

daniandtheweb commented 1 year ago

Thanks @SlyEcho for the info. I also think that a command line argument would be the best but I guess that for now I can just close the issue.