Open BurnyLlama opened 1 month ago
If I try to use v0.14.0 (same options) I instead get this:
That i was get too, but only at builded tabby, when i am compiling 0.14.0 by my self it's works
By the way, ROCm will work at RX6600.
Tabby only supports the use of a single GPU. To utilize multiple GPUs, you can initiate multiple Tabby instances and set CUDA_VISIBLE_DEVICES (for cuda) or HIP_VISIBLE_DEVICES (for rocm) accordingly. You can use the HSA_OVERRIDE_GFX_VERSION variable if there is a similar GPU that is supported by ROCm you can set it to that. For example for RDNA2 you can set it to 10.3.0 and to 11.0.0 for RDNA3.
Right! I never tried to build v0.14.0
myself. I will see if I have time to try that later. If so, I will come back with the results.
I realized I didn't compile for ROCm before, only for Vulkan. When I try to compile with --features=rocm
:
error: failed to run custom build command for `llama-cpp-server v0.14.0 (/tmp/tabby-src/crates/llama-cpp-server)`
Caused by:
process didn't exit successfully: `/tmp/tabby-src/target/release/build/llama-cpp-server-1b3738a0281592c2/build-script-build` (exit status: 101)
--- stdout
CMAKE_TOOLCHAIN_FILE_x86_64-unknown-linux-gnu = None
CMAKE_TOOLCHAIN_FILE_x86_64_unknown_linux_gnu = None
HOST_CMAKE_TOOLCHAIN_FILE = None
CMAKE_TOOLCHAIN_FILE = None
CMAKE_GENERATOR_x86_64-unknown-linux-gnu = None
CMAKE_GENERATOR_x86_64_unknown_linux_gnu = None
HOST_CMAKE_GENERATOR = None
CMAKE_GENERATOR = None
CMAKE_PREFIX_PATH_x86_64-unknown-linux-gnu = None
CMAKE_PREFIX_PATH_x86_64_unknown_linux_gnu = None
HOST_CMAKE_PREFIX_PATH = None
CMAKE_PREFIX_PATH = None
CMAKE_x86_64-unknown-linux-gnu = None
CMAKE_x86_64_unknown_linux_gnu = None
HOST_CMAKE = None
CMAKE = None
running: cd "/tmp/tabby-src/target/release/build/llama-cpp-server-40e8fadd415341a4/out/build" && CMAKE_PREFIX_PATH="" "cmake" "/tmp/tabby-src/crates/llama-cpp-server/./llama.cpp" "-DLLAMA_NATIVE=OFF" "-DBUILD_SHARED_LIBS=OFF" "-DINS_ENB=ON" "-DLLAMA_HIPBLAS=ON" "-DCMAKE_C_COMPILER=/opt/rocm/llvm/bin/clang" "-DCMAKE_CXX_COMPILER=/opt/rocm/llvm/bin/clang++" "-DAMDGPU_TARGETS=gfx803;gfx900;gfx906:xnack-;gfx908:xnack-;gfx90a:xnack+;gfx90a:xnack-;gfx940;gfx941;gfx942;gfx1010;gfx1012;gfx1030;gfx1031;gfx1100;gfx1101;gfx1102;gfx1103" "-DCMAKE_INSTALL_PREFIX=/tmp/tabby-src/target/release/build/llama-cpp-server-40e8fadd415341a4/out" "-DCMAKE_C_FLAGS= -ffunction-sections -fdata-sections -fPIC -m64" "-DCMAKE_CXX_FLAGS= -ffunction-sections -fdata-sections -fPIC -m64" "-DCMAKE_ASM_FLAGS= -ffunction-sections -fdata-sections -fPIC -m64" "-DCMAKE_ASM_COMPILER=/usr/bin/cc" "-DCMAKE_BUILD_TYPE=Release"
-- The C compiler identification is unknown
-- The CXX compiler identification is unknown
-- Configuring incomplete, errors occurred!
--- stderr
CMake Error at CMakeLists.txt:2 (project):
The CMAKE_C_COMPILER:
/opt/rocm/llvm/bin/clang
is not a full path to an existing compiler tool.
Tell CMake where to find the compiler by setting either the environment
variable "CC" or the CMake cache entry CMAKE_C_COMPILER to the full path to
the compiler, or to the compiler name if it is in the PATH.
CMake Error at CMakeLists.txt:2 (project):
The CMAKE_CXX_COMPILER:
/opt/rocm/llvm/bin/clang++
is not a full path to an existing compiler tool.
Tell CMake where to find the compiler by setting either the environment
variable "CXX" or the CMake cache entry CMAKE_CXX_COMPILER to the full path
to the compiler, or to the compiler name if it is in the PATH.
thread 'main' panicked at /home/USER/.cargo/registry/src/index.crates.io-6f17d22bba15001f/cmake-0.1.50/src/lib.rs:1098:5:
command did not execute successfully, got: exit status: 1
build script failed, must exit now
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
I use Gentoo and I should (as far as I know) have ROCm installed, but maybe I am missing something... I'll have to look into this more.
(This happens both on v0.14.0
and v0.15.0-r3
...)
A bit of a hacky solution that at least gets Tabby to compile:
sudo mkdir /opt/rocm
sudo ln -sv /usr/lib/llvm/18 /opt/rocm/llvm
However, besides seeming slower to generate responses in the chat, there's no difference. It still seems to run on the CPU when using:
HSA_OVERRIDE_GFX_VERSION=10.3.0 cargo run --features rocm --release serve --model DeepseekCoder-1.3B --chat-model Qwen2-1.5B-Instruct --device rocm
I also tried adding HCC_AMDGPU_TARGET=gfx1032
to specify using my GPU and not my CPU. I don't know if that is how I am supposed to use that environment variable, but it did not work.
The fix for rocm is merged, the vulkan fix is probably similar (I haven't tested it): https://github.com/TabbyML/tabby/issues/2810#issuecomment-2283356626
Describe the bug
(This might be me doing something wrong, and not a bug!)
I can't seem to get Tabby to run using my GPU (Radeon 6600 XT). Not with ROC, (which I believe is unsupported for my device) nor with Vulkan — which I believe should be supported?
Whenever I run tabby (
serve --model DeepseekCoder-1.3B --chat-model Qwen2-1.5B-Instruct --device vulkan
) and go into the Web UI to ask something I check my CPU and GPU usage (usingbtop
andamdgpu_top
respectively) I see my CPU usage spiking and almost no effect on the GPU. (This is when runningv0.15.0-r2
or compilingv0.15.0-r3
myself).If I try to use
v0.14.0
(same options) I instead get this:Information about your version
See above.
Information about your GPU
From
vulkaninfo
:From
rocminfo
(if it helps):Additional context
Full terminal output when starting Tabby (
v0.15.0-r3
, compiled myself):