FileNotFoundError: Could not find module 'J:\AI\text-generation-webui\installer_files\env\Lib\site-packages\llama_cpp_cuda\lib\llama.dll'

allrobot commented 3 months ago

Describe the bug

Unable to load the model normally, but llama-cpp-python can load the model without issues. I don't know why llama.cpp from text-generation-webui cannot load the model, showing an error that llama.dll is not found, even though this DLL does actually exist.

Is there an existing issue for this?

[X] I have searched the existing issues

Reproduction

git clone https://github.com/oobabooga/text-generation-webui
cd text-generation-webui
pip install -r requirements.txt
python server.py

Download the TheBloke/CausalLM-14B-GGUF model, then switch to the Model interface and select llama.cpp on http://127.0.0.1:7860/. However, it throws an error after loading the model.

Screenshot

explorer_CKZkQ5wqRY

Logs

Traceback (most recent call last):

File "J:\AI\text-generation-webui\installer_files\env\Lib\site-packages\llama_cpp_cuda\llama_cpp.py", line 75, in _load_shared_library

return ctypes.CDLL(str(_lib_path), **cdll_args)  # type: ignore

       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "J:\AI\text-generation-webui\installer_files\env\Lib\ctypes_init_.py", line 376, in init

self._handle = _dlopen(self._name, mode)

               ^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: Could not find module 'J:\AI\text-generation-webui\installer_files\env\Lib\site-packages\llama_cpp_cuda\lib\llama.dll' (or one of its dependencies). Try using the full path with constructor syntax.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "J:\AI\text-generation-webui\modules\ui_model_menu.py", line 248, in load_model_wrapper

shared.model, shared.tokenizer = load_model(selected_model, loader)

                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "J:\AI\text-generation-webui\modules\models.py", line 94, in load_model

output = load_func_map[loader](model_name)

         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "J:\AI\text-generation-webui\modules\models.py", line 275, in llamacpp_loader

model, tokenizer = LlamaCppModel.from_pretrained(model_file)

                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "J:\AI\text-generation-webui\modules\llamacpp_model.py", line 38, in from_pretrained

Llama = llama_cpp_lib().Llama

        ^^^^^^^^^^^^^^^
File "J:\AI\text-generation-webui\modules\llama_cpp_python_hijack.py", line 42, in llama_cpp_lib

return_lib = importlib.import_module(lib_name)

             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "J:\AI\text-generation-webui\installer_files\env\Lib\importlib_init_.py", line 126, in import_module

return _bootstrap._gcd_import(name[level:], package, level)

       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 1206, in _gcd_import

File "", line 1178, in _find_and_load

File "", line 1149, in _find_and_load_unlocked

File "", line 690, in _load_unlocked

File "", line 940, in exec_module

File "", line 241, in _call_with_frames_removed

File "J:\AI\text-generation-webui\installer_files\env\Lib\site-packages\llama_cpp_cuda_init_.py", line 1, in

from .llama_cpp import *
File "J:\AI\text-generation-webui\installer_files\env\Lib\site-packages\llama_cpp_cuda\llama_cpp.py", line 88, in

_lib = _load_shared_library(_lib_base_name)

       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "J:\AI\text-generation-webui\installer_files\env\Lib\site-packages\llama_cpp_cuda\llama_cpp.py", line 77, in _load_shared_library

raise RuntimeError(f"Failed to load shared library '{_lib_path}': {e}")
RuntimeError: Failed to load shared library 'J:\AI\text-generation-webui\installer_files\env\Lib\site-packages\llama_cpp_cuda\lib\llama.dll': Could not find module 'J:\AI\text-generation-webui\installer_files\env\Lib\site-packages\llama_cpp_cuda\lib\llama.dll' (or one of its dependencies). Try using the full path with constructor syntax.

System Info

OS: WIN10 enterprise 21H2 LTSC
GPU: GTX3080 10G 

+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 555.99                 Driver Version: 555.99         CUDA Version: 12.5     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                  Driver-Model | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 3080      WDDM  |   00000000:01:00.0  On |                  N/A |
|  0%   38C    P8             26W /  320W |    1925MiB /  10240MiB |      1%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A       824    C+G   C:\Windows\explorer.exe                     N/A      |
|    0   N/A  N/A      1572    C+G   ...\CentBrowser\Application\chrome.exe      N/A      |
|    0   N/A  N/A      1916    C+G   C:\Windows\System32\dwm.exe                 N/A      |
|    0   N/A  N/A      4664    C+G   ...CBS_cw5n1h2txyewy\TextInputHost.exe      N/A      |
|    0   N/A  N/A      9104    C+G   ...ys\WinUI3Apps\PowerToys.Peek.UI.exe      N/A      |
|    0   N/A  N/A     10256    C+G   ....Search_cw5n1h2txyewy\SearchApp.exe      N/A      |
|    0   N/A  N/A     12636    C+G   C:\Program Files\SiYuan\SiYuan.exe          N/A      |
|    0   N/A  N/A     19172    C+G   ...nipaste-2.6.6-Beta-x64\Snipaste.exe      N/A      |
|    0   N/A  N/A     22152    C+G   ...5n1h2txyewy\ShellExperienceHost.exe      N/A      |
|    0   N/A  N/A     22484    C+G   ...\PowerToys\PowerToys.FancyZones.exe      N/A      |
|    0   N/A  N/A     25756    C+G   ...siveControlPanel\SystemSettings.exe      N/A      |
|    0   N/A  N/A     30780    C+G   ...PowerToys\PowerToys.PowerAccent.exe      N/A      |
|    0   N/A  N/A     32156    C+G   ...\Huorong\Sysdiag\bin\HipsDaemon.exe      N/A      |
|    0   N/A  N/A     39836    C+G   ...werToys\PowerToys.ColorPickerUI.exe      N/A      |
|    0   N/A  N/A     42164    C+G   ...ty Edition 2024.1\bin\pycharm64.exe      N/A      |
+-----------------------------------------------------------------------------------------+

roman-dj-95 commented 3 months ago

Had exactly the same error after updating text-generation-webui. Updating CUDA from 11 to 12 (latest) solved this problem for me.

allrobot commented 3 months ago

Had exactly the same error after updating text-generation-webui. Updating CUDA from 11 to 12 (latest) solved this problem for me.

So, I forgot to add the CUDA 12 path to the environment variables, and after adding it, it showed that the loading was successful.

HavingIssues1088 commented 2 weeks ago

Hello, I was wondering allrobot, if you can tell me specifically how you solved this issue. I appear to be having it as well and everything I have tried has proved fruitless. I'm not very proficient in these matters but everything seems to be up to date and despite beginning from scratch my webui still gives the very error you were having.

Could you walk me through adding the CUDA 12 path to the environment variables? Simply updating to 12 did not solve the issue. Thanks in advance for any assistance you can give.

oobabooga / text-generation-webui