Something is wrong with 0.2.5 - chatdocs download command

[Ananderz]

Hi @marella

Something broke with 0.2.5. I have tried many different ways to get it to work.

When I run 0.2.4 everything works flawlessly.

With 0.2.5 something goes wrong any time I do chatdocs download, it does not seem to respect changes to chatdocs.yml

When I go into the site-packages folder and change to the correct model file path it give me the error that ctransformers.dll was not found.

Anyone able to fix this?

[marella]

Hi, please share the full error message you are getting. Are you using it with GPU? Can you please try running it without specifying gpu_layers and see if it works.

Can you please run the following commands and share their output:

 nvidia-smi

 pip show ctransformers

[Ananderz]

Hi @marella

Here is nvidia-smi: `nvidia-smi Mon Jul 31 12:54:59 2023 +---------------------------------------------------------------------------------------+	NVIDIA-SMI 536.40 Driver Version: 536.40 CUDA Version: 12.2
-----------------------------------------+----------------------+----------------------+
GPU Name TCC/WDDM	Bus-Id Disp.A	Volatile Uncorr. ECC
Fan Temp Perf Pwr:Usage/Cap	Memory-Usage	GPU-Util Compute M.
		MIG M.
=========================================+======================+======================
0 NVIDIA GeForce RTX 3060 WDDM	00000000:4B:00.0 On	N/A
30% 32C P8 11W / 170W	9372MiB / 12288MiB	2% Default
		N/A

+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+	Processes:
GPU GI CI PID Type Process name GPU Memory
ID ID Usage
=======================================================================================
0 N/A N/A 1116 C+G ...(x86)\GIGABYTE\AppCenter\ApCent.exe N/A
0 N/A N/A 1384 C+G ...siveControlPanel\SystemSettings.exe N/A
0 N/A N/A 2108 C+G ...hunderbolt Software\Thunderbolt.exe N/A
0 N/A N/A 3484 C+G ...oogle\Chrome\Application\chrome.exe N/A
0 N/A N/A 8524 C+G C:\Windows\explorer.exe N/A
0 N/A N/A 8988 C+G ...\Programs\signal-desktop\Signal.exe N/A
0 N/A N/A 8992 C+G ...crosoft\Edge\Application\msedge.exe N/A
0 N/A N/A 9712 C+G ...2txyewy\StartMenuExperienceHost.exe N/A
0 N/A N/A 10296 C+G ...5n1h2txyewy\ShellExperienceHost.exe N/A
0 N/A N/A 10520 C+G ....Search_cw5n1h2txyewy\SearchApp.exe N/A
0 N/A N/A 13580 C+G ...GeForce Experience\NVIDIA Share.exe N/A
0 N/A N/A 13940 C ...Programs\Python\Python39\python.exe N/A
0 N/A N/A 14036 C+G ...CBS_cw5n1h2txyewy\TextInputHost.exe N/A
0 N/A N/A 15592 C+G ...al\Discord\app-1.0.9015\Discord.exe N/A
0 N/A N/A 18068 C+G D:\Visual Studio Code\Code.exe N/A
0 N/A N/A 18360 C+G ...t.LockApp_cw5n1h2txyewy\LockApp.exe N/A
0 N/A N/A 18412 C+G ..._x64__kzf8qxf38zg5c\Skype\Skype.exe N/A
0 N/A N/A 22308 C+G ..._8wekyb3d8bbwe\Microsoft.Photos.exe N/A
0 N/A N/A 24304 C+G ...ekyb3d8bbwe\PhoneExperienceHost.exe N/A

+---------------------------------------------------------------------------------------+`

Here is pip show ctransformers: Name: ctransformers Version: 0.2.11 Summary: Python bindings for the Transformer models implemented in C/C++ using GGML library. Home-page: https://github.com/marella/ctransformers Author: Ravindra Marella Author-email: mv.ravindra007@gmail.com License: MIT Location: c:\users\test\appdata\local\programs\python\python39\lib\site-packages Requires: huggingface-hub Required-by: chatdocs

Here is the error: ╭───────────────────────── Traceback (most recent call last) ──────────────────────────╮│ C:\Users\test\AppData\Local\Programs\Python\Python39\lib\site-packages\chatdocs-0. ││ 2.5-py3.9.egg\chatdocs\main.py:26 in download ││ ││ 23 │ from .download import download ││ 24 │ ││ 25 │ config = get_config(config) ││ ❱ 26 │ download(config=config) ││ 27 ││ 28 ││ 29 @app.command() ││ ││ C:\Users\test\AppData\Local\Programs\Python\Python39\lib\site-packages\chatdocs-0. ││ 2.5-py3.9.egg\chatdocs\download.py:10 in download ││ ││ 7 def download(config: Dict[str, Any]) -> None: ││ 8 │ config = {config, "download": True} ││ 9 │ get_embeddings(config) ││ ❱ 10 │ get_llm(config) ││ 11 ││ ││ C:\Users\test\AppData\Local\Programs\Python\Python39\lib\site-packages\chatdocs-0. ││ 2.5-py3.9.egg\chatdocs\llms.py:73 in get_llm ││ ││ 70 │ if config["llm"] == "ctransformers": ││ 71 │ │ config = {config["ctransformers"]} ││ 72 │ │ config = merge(config, {"config": {"local_files_only": local_files_only ││ ❱ 73 │ │ llm = CTransformers(callbacks=callbacks, config) ││ 74 │ elif config["llm"] == "gptq": ││ 75 │ │ llm = get_gptq_llm(config) ││ 76 │ else: ││ ││ C:\Users\test\AppData\Local\Programs\Python\Python39\lib\site-packages\langchain\l ││ oad\serializable.py:64 in init ││ ││ 61 │ _lc_kwargs = PrivateAttr(default_factory=dict) ││ 62 │ ││ 63 │ def init(self, kwargs: Any) -> None: ││ ❱ 64 │ │ super().init(**kwargs) ││ 65 │ │ self._lc_kwargs = kwargs ││ 66 │ ││ 67 │ def to_json(self) -> Union[SerializedConstructor, SerializedNotImplemented ││ ││ C:\Users\test\Downloads\chatdocs-main\chatdocs-main\pydantic\main.py:339 in ││ pydantic.main.BaseModel.init ││ ││ [Errno 2] No such file or directory: ││ 'C:\Users\test\Downloads\chatdocs-main\chatdocs-main\pydantic\main.py' ││ ││ C:\Users\test\Downloads\chatdocs-main\chatdocs-main\pydantic\main.py:1102 in ││ pydantic.main.validate_model ││ ││ [Errno 2] No such file or directory: ││ 'C:\Users\test\Downloads\chatdocs-main\chatdocs-main\pydantic\main.py' ││ ││ C:\Users\test\AppData\Local\Programs\Python\Python39\lib\site-packages\langchain\l ││ lms\ctransformers.py:70 in validate_environment ││ ││ 67 │ │ │ ) ││ 68 │ │ ││ 69 │ │ config = values["config"] or {} ││ ❱ 70 │ │ values["client"] = AutoModelForCausalLM.from_pretrained( ││ 71 │ │ │ values["model"], ││ 72 │ │ │ model_type=values["model_type"], ││ 73 │ │ │ model_file=values["model_file"], ││ ││ c:\users\test\appdata\local\programs\python\python39\lib\site-packages\ctransforme ││ rs-0.2.13-py3.9.egg\ctransformers\hub.py:157 in from_pretrained ││ ││ 154 │ │ │ │ local_files_only=local_files_only, ││ 155 │ │ │ ) ││ 156 │ │ ││ ❱ 157 │ │ return LLM( ││ 158 │ │ │ model_path=model_path, ││ 159 │ │ │ model_type=model_type, ││ 160 │ │ │ config=config.config, ││ ││ c:\users\test\appdata\local\programs\python\python39\lib\site-packages\ctransforme ││ rs-0.2.13-py3.9.egg\ctransformers\llm.py:206 in init ││ ││ 203 │ │ if not Path(model_path).is_file(): ││ 204 │ │ │ raise ValueError(f"Model path '{model_path}' doesn't exist.") ││ 205 │ │ ││ ❱ 206 │ │ self._lib = load_library(lib, cuda=config.gpu_layers > 0) ││ 207 │ │ self._llm = self._lib.ctransformers_llm_create( ││ 208 │ │ │ model_path.encode(), ││ 209 │ │ │ model_type.encode(), ││ ││ c:\users\test\appdata\local\programs\python\python39\lib\site-packages\ctransforme ││ rs-0.2.13-py3.9.egg\ctransformers\llm.py:102 in load_library ││ ││ 99 │ │ os.add_dll_directory(os.path.join(os.environ["CUDA_PATH"], "bin")) ││ 100 │ ││ 101 │ path = find_library(path, cuda=cuda) ││ ❱ 102 │ lib = CDLL(path) ││ 103 │ ││ 104 │ lib.ctransformers_llm_create.argtypes = [ ││ 105 │ │ c_char_p, # model_path ││ ││ C:\Users\test\AppData\Local\Programs\Python\Python39\lib\ctypes__init.py:374 in ││ init__ ││ ││ 371 │ │ self._FuncPtr = _FuncPtr ││ 372 │ │ ││ 373 │ │ if handle is None: ││ ❱ 374 │ │ │ self._handle = _dlopen(self._name, mode) ││ 375 │ │ else: ││ 376 │ │ │ self._handle = handle ││ 377 │╰──────────────────────────────────────────────────────────────────────────────────────╯FileNotFoundError: Could not find module 'C:\Users\test\AppData\Local\Programs\Python\Python39\Lib\site-packages\ctransformers-0.2.13-py3.9.egg\ctransformers\lib\cuda\ctransformers.dll' (or one of its dependencies).Try using the full path with constructor syntax.

The ctransformers.dll is in the specified folder and 0.2.4 works flawlessly. However 0.2.5 does not seem to work

[marella]

This can happen if there are multiple versions of CUDA installed. In the latest version (0.2.21) of ctransformers I have simplified CUDA installation. Please update ctransformers and try again:

pip install 'ctransformers>=0.2.21'

Install CUDA libraries using:

pip install ctransformers[cuda]

[Ananderz]

This does not fix the error unfortunately :(

[marella]

Can you please share the full error message and also the output of following command:

pip show ctransformers nvidia-cuda-runtime-cu12 nvidia-cublas-cu12

Another option is to build it from source by running the commands provided in README here.

[Ananderz]

@marella WARNING: Package(s) not found: nvidia-cublas-cu12, nvidia-cuda-runtime-cu12 Name: ctransformers Version: 0.2.11 Summary: Python bindings for the Transformer models implemented in C/C++ using GGML library. Home-page: https://github.com/marella/ctransformers Author: Ravindra Marella Author-email: mv.ravindra007@gmail.com License: MIT Location: X (removed this) Requires: huggingface-hub Required-by: chatdocs

[marella]

Hi, you are using an old version ctransformers and haven't installed the CUDA libraries. Please run the following commands:

pip install 'ctransformers>=0.2.24'
pip install ctransformers[cuda]

Please double check that you have run these commands in the correct environment where you have installed chatdocs.

[Ananderz]

Hi, you are using an old version ctransformers and haven't installed the CUDA libraries. Please run the following commands:

pip install 'ctransformers>=0.2.24'
pip install ctransformers[cuda]

Please double check that you have run these commands in the correct environment where you have installed chatdocs.

Thank you!!! it works! I updated to 0.2.26 and ran the ctransformers[cuda] and now the models don't take up all my gpu power, it runs at about 50%