AMD

[pierceglenn12345]

I am not sure if this is open in another issue mentioned previously, or at least I haven't found it. I have an AMD Radeon graphics card with 8g VRAM. I should be able to run 7B gptq models okay, but I am getting an error on run_localGPT.py saying:

RuntimeError: Found no NVIDIA driver on your system. Please check that you have an NVIDIA GPU and installed a driver from http://www.nvidia.com/Download/index.aspx

I tried using the flag --device_type for both hip and opencl, but that gives me an error:

RuntimeError: PyTorch is not linked with support for hip devices.

This stackoverflow mentions this: https://stackoverflow.com/questions/64523498/can-you-accelerate-torch-dl-training-on-anything-other-than-cuda-like-hip-or

and says there is support for hip in pytorch. maybe they are wrong? Should I be downloading something to get this to work.

Any tips would be much appreciated!

[rcantada]

Perhaps the installation instructions for text-generation-webui at https://github.com/oobabooga/text-generation-webui can provide a clue. They include installing the following with rocm among other things: pytorch for AMD--linux only, but I don't know if torchaudio is needed for localGPT

pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm5.4.2

auto-gptq

pip install auto-gptq==0.2.2 --extra-index-url https://huggingface.github.io/autogptq-index/whl/rocm542/

and llama-cpp

CMAKE_ARGS="-DLLAMA_HIPBLAS=on" FORCE_CMAKE=1 pip install llama-cpp-python==0.1.83 --no-cache-dir

there may be others in the requirements list that would need rocm though... perhaps others can give a hint.

[the-snowwhite]

I have a AMD RX 7900 XTX Gpu and was able to get this to install after updating requirements.txt utilizing this PR: https://github.com/PromtEngineer/localGPT/pull/555 . Then I finally got it to run (with cuda) after finding this thread: https://github.com/ggerganov/llama.cpp/issues/3320 . I have the amdgpu driver installed with:

amdgpu-install --usecase=rocm --rocmrelease=5.7.0

(without the opencl) And llama.cpp-python installed with:

CMAKE_ARGS="-DLLAMA_HIPBLAS=on -DCMAKE_CXX_FLAGS='-fPIC' -DAMDGPU_TARGETS=gfx1100" FORCE_CMAKE=1 CC=/opt/rocm/llvm/bin/clang CXX=/opt/rocm/llvm/bin/clang++ pip install llama-cpp-python --no-cache-dir --force-reinstall

Pytorch is installed with:

pip3 install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/rocm5.7

[macvsogjc]

I have a AMD RX 7900 XTX Gpu and was able to get this to install after updating requirements.txt utilizing this PR: #555 . Then I finally got it to run (with cuda) after finding this thread: ggerganov/llama.cpp#3320 . I have the amdgpu driver installed with:

amdgpu-install --usecase=rocm --rocmrelease=5.7.0

(without the opencl) And llama.cpp-python installed with:

CMAKE_ARGS="-DLLAMA_HIPBLAS=on -DCMAKE_CXX_FLAGS='-fPIC' -DAMDGPU_TARGETS=gfx1100" FORCE_CMAKE=1 CC=/opt/rocm/llvm/bin/clang CXX=/opt/rocm/llvm/bin/clang++ pip install llama-cpp-python --no-cache-dir --force-reinstall

Pytorch is installed with:

pip3 install --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/rocm5.7

I'm a complete beginner here, but really keen to try and get this (and other LLMs) running on my RX 6600XT.

Are you able to help with a bit of a step-by-step guide on this please?

Do I need to update or change anything in the original requirements.txt?

Do I run those console commands you've listed above within the conda environment I've created for localgpt? I'm abit confused about the 'without the opencl' remark and llama.cpp.

Thanks in advance!

[the-snowwhite]

@johnsmithtetley As PR: https://github.com/PromtEngineer/localGPT/pull/555 has been merged since my post there is now nothing you have to change in requirements.txt if you use the updated repository.

The "without opencl" means that you install the amd driver (.deb downloaded from AMD's site) without installing opencl support. (with the first bash command listed in my previous post)(after installing the .deb package). The amd GPU driver is a global install for your whole system.

Yes The 2 other commands are run within your local conda environment, however you need to change:

-DAMDGPU_TARGETS=gfx1100"

to:

-DAMDGPU_TARGETS=gfx1032"

To mach the RX 6600XT GPU. for the llama.cpp-python install mod.

The "without opencl" also means that you do not use the flag --device_type for hip or opencl. But use the default cuda as device_type target as specified in the READ.ME.