cannot load llama_cpp\llama.dll' of the rocm wheels. but the installation is suceed.

[sorasoras]

2023-11-24 03:23:15 SORANET main[11664] WARNING Auth is disabled! 2023-11-24 03:23:15 SORANET main[11664] INFO Current server config: Server(listen: 0.0.0.0:5000, auth: None:None) 2023-11-24 03:23:15 SORANET main[11664] INFO Current model config: SakuraModelConfig(model_name_or_path='ggml-model-4bit.gguf', use_gptq_model=False, trust_remote_code=True, text_length=512, llama=False, llama_cpp=True, use_gpu=True, n_gpu_layers=0, model_name=None, model_quant=None, model_version='0.8') Traceback (most recent call last): File "C:\Users\shing\AppData\Local\Programs\Python\Python311\Lib\site-packages\llama_cpp\llama_cpp.py", line 69, in _load_shared_library return ctypes.CDLL(str(_lib_path), **cdll_args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\shing\AppData\Local\Programs\Python\Python311\Lib\ctypes__init.py", line 376, in init__ self._handle = _dlopen(self._name, mode) ^^^^^^^^^^^^^^^^^^^^^^^^^ FileNotFoundError: Could not find module 'C:\Users\shing\AppData\Local\Programs\Python\Python311\Lib\site-packages\llama_cpp\llama.dll' (or one of its dependencies). Try using the full path with constructor syntax.

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "W:\llama\Sakura-13B-Galgame\server.py", line 122, in state.init_model(cfg) File "W:\llama\Sakura-13B-Galgame\utils\state.py", line 21, in init_model sakura_model = SakuraModel(*args, *kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "W:\llama\Sakura-13B-Galgame\utils\model.py", line 127, in init (tokenizer, model) = load_model(cfg) ^^^^^^^^^^^^^^^ File "W:\llama\Sakura-13B-Galgame\utils\model.py", line 71, in load_model from llama_cpp import Llama File "C:\Users\shing\AppData\Local\Programs\Python\Python311\Lib\site-packages\llama_cpp__init__.py", line 1, in from .llama_cpp import File "C:\Users\shing\AppData\Local\Programs\Python\Python311\Lib\site-packages\llama_cpp\llama_cpp.py", line 82, in _lib = _load_shared_library(_lib_base_name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\shing\AppData\Local\Programs\Python\Python311\Lib\site-packages\llama_cpp\llama_cpp.py", line 71, in _load_shared_library raise RuntimeError(f"Failed to load shared library '{_lib_path}': {e}") RuntimeError: Failed to load shared library 'C:\Users\shing\AppData\Local\Programs\Python\Python311\Lib\site-packages\llama_cpp\llama.dll': Could not find module 'C:\Users\shing\AppData\Local\Programs\Python\Python311\Lib\site-packages\llama_cpp\llama.dll' (or one of its dependencies). Try using the full path with constructor syntax.

What dependency do i missed? thanks.

[jllllll]

Hard to say for certain. The main external requirement for the ROCm version is ROCm itself.

Windows ROCm is here: https://www.amd.com/en/developer/rocm-hub/hip-sdk.html GPU compatibility chart: https://rocm.docs.amd.com/en/docs-5.5.1/release/windows_support.html

[sorasoras]

Hard to say for certain. The main external requirement for the ROCm version is ROCm itself.

Windows ROCm is here: https://www.amd.com/en/developer/rocm-hub/hip-sdk.html GPU compatibility chart: https://rocm.docs.amd.com/en/docs-5.5.1/release/windows_support.html

Do Rocm require specific GPU drivers. I am using the gaming driver. that might be the issue. The SDK come with a bundle optional driver through

[sorasoras]

reinstall hip sdk with its driver as well, it does not work. I got a 7900XTX so i think hardware is compatible

[jllllll]

I honestly don't know what the issue is. The error is simply too generic to know how to fix it. It is missing something that it requires, but that could be many things or it could simply be unable to find what it requires, which can be caused by many things.

[sorasoras]

I honestly don't know what the issue is. The error is simply too generic to know how to fix it. It is missing something that it requires, but that could be many things or it could simply be unable to find what it requires, which can be caused by many things.

I was able to get it working by adding Rocm/bin to the ENV, but the speed is extremely slow. By replacing DLL that I compiled from source, I was able to run a 4bit 13B model at 55token/s on my 7900XTX. That's great.