Llama-cpp-python compilation for text-generation-webui using LLAMA_HIPBLAS=on for rocm

[mpwx007]

For those who have some problems compiling llama-cpp-python with ROCm 6.0 (https://github.com/abetlen/llama-cpp-python), here is how I did it:

• Follow all the tutorials here for installing text-generation-webui.
• Execute the following code on the native terminal (I got an error when I compiled with tabby...) in accordance with the access paths and you graphics card (and I recommand you, to add this in your .bashrc):

  1. export HIP_PLATFORM=amd
     export HIP_PATH=/opt/rocm-6.0.2/
     export PATH=$PATH:/opt/rocm/bin:/opt/rocm-6.0.2/bin
     export HSA_OVERRIDE_GFX_VERSION=11.0.0
     export PYTORCH_ROCM_ARCH="gfx1100"
     export HIP_VISIBLE_DEVICES=0
     export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/rocm/lib:/opt/rocm-6.0.2/lib

     _(/!\ your HIP_VISIBLEDEVICES can be 1, check it if it not working) [check if there is no error with the following command: /opt/rocm/bin/hipconfig --full] 2: sudo apt-get install libstdc++-12-dev libstdc++-12-doc 3: conda activate textgen 4: CMAKE_ARGS="-D LLAMA_HIPBLAS=ON -D CMAKE_C_COMPILER=/opt/rocm-6.0.2/llvm/bin/clang -D CMAKE_CXX_COMPILER=/opt/rocm-6.0.2/llvm/bin/clang++ -D CMAKE_PREFIX_PATH=/opt/rocm-6.0.2 -D AMDGPU_TARGETS=gfx1100" pip install llama-cpp-python==0.2.75 --no-deps --force-reinstall --no-cache NB: dont forget to change gfx1100 if you have other graphic card than RX 7900 serie

NB: If your version is different, edit it accordingly.

Hope this help you !

[nktice]

I went and tried this... and found another method making it from source.

conda activate textgen
# remove old version if needed...
pip uninstall llama_cpp_python
pip uninstall llama_cpp_python_cuda

mkdir ~/text-generation-webui/repositories/
cd ~/text-generation-webui/repositories/
git clone  --recurse-submodules  https://github.com/abetlen/llama-cpp-python.git
cd llama-cpp-python
pip install .