I am trying to dockerize a LLM (Mistral) on my local laptop using Ctransformers. For some reason I am noticing a weird error below. However, the same is not happening If I run the model locally in my laptop. Could you let me know if there is any compatibility issues with mistral LLM and ctransformers. I am using python version 3.12 and I tried with all other version. Nothing is working. Your help will be much appreciated.
Traceback (most recent call last):
File "/usr/src/app/app.py", line 9, in
llm = CTransformers(
^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/pydantic/v1/main.py", line 339, in init
values, fields_set, validation_error = validate_model(__pydantic_self.class, data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/pydantic/v1/main.py", line 1100, in validatemodel
values = validator(cls, values)
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/langchain_community/llms/ctransformers.py", line 72, in validate_environment
values["client"] = AutoModelForCausalLM.from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/ctransformers/hub.py", line 173, in from_pretrained
return LLM(
^^^^
File "/usr/local/lib/python3.12/site-packages/ctransformers/llm.py", line 237, in init
self._lib = load_library(lib, cuda=config.gpu_layers > 0)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/ctransformers/llm.py", line 124, in load_library
lib = CDLL(path)
^^^^^^^^^^
File "/usr/local/lib/python3.12/ctypes/init.py", line 379, in init__
self._handle = _dlopen(self._name, mode)
^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: /usr/local/lib/python3.12/site-packages/ctransformers/lib/basic/libctransformers.so: cannot open shared object file: No such file or directory
Ajayvenki
commented
1 month ago
I am trying to dockerize a LLM (Mistral) on my local laptop using Ctransformers. For some reason I am noticing a weird error below. However, the same is not happening If I run the model locally in my laptop. Could you let me know if there is any compatibility issues with mistral LLM and ctransformers. I am using python version 3.12 and I tried with all other version. Nothing is working. Your help will be much appreciated.
Traceback (most recent call last): File "/usr/src/app/app.py", line 9, in
llm = CTransformers(
^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/pydantic/v1/main.py", line 339, in init
values, fields_set, validation_error = validate_model(__pydantic_self.class, data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/pydantic/v1/main.py", line 1100, in validatemodel
values = validator(cls, values)
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/langchain_community/llms/ctransformers.py", line 72, in validate_environment
values["client"] = AutoModelForCausalLM.from_pretrained(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/ctransformers/hub.py", line 173, in from_pretrained
return LLM(
^^^^
File "/usr/local/lib/python3.12/site-packages/ctransformers/llm.py", line 237, in init
self._lib = load_library(lib, cuda=config.gpu_layers > 0)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/ctransformers/llm.py", line 124, in load_library
lib = CDLL(path)
^^^^^^^^^^
File "/usr/local/lib/python3.12/ctypes/init.py", line 379, in init__
self._handle = _dlopen(self._name, mode)
^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: /usr/local/lib/python3.12/site-packages/ctransformers/lib/basic/libctransformers.so: cannot open shared object file: No such file or directory
Solution i have tried: I have tried this after a research and it dint work as well - https://github.com/marella/ctransformers/issues/9#issuecomment-1557140200
I gave a latest comment in the JIRA, could u help