search

neuml / txtai

💡 All-in-one open-source embeddings database for semantic search, LLM orchestration and language model workflows
https://neuml.github.io/txtai
Apache License 2.0
9.24k stars 596 forks source link

Can't read locally downloaded model with `LLM('/local/path/model')` #639

Closed 4l1fe closed 10 months ago

4l1fe commented 10 months ago

In regard to our conversation #635, i've tried to download manually and automatically a model, then passed a path of it to the class LLM and got the error

>>> llm = LLM('/data/models/tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf', method='llama.cpp')
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/transformers/configuration_utils.py", line 729, in _get_config_dict
    config_dict = cls._dict_from_json_file(resolved_config_file)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/transformers/configuration_utils.py", line 827, in _dict_from_json_file
    text = reader.read()
           ^^^^^^^^^^^^^
  File "<frozen codecs>", line 322, in decode
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc9 in position 8: invalid continuation byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.11/site-packages/txtai/pipeline/text/llm.py", line 18, in __init__
    super().__init__(self.task(path, task, **kwargs), path if path else "google/flan-t5-base", quantize, gpu, model, **kwargs)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/txtai/pipeline/text/llm.py", line 103, in task
    task = Models.task(path, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/txtai/models/models.py", line 239, in task
    config = AutoConfig.from_pretrained(path, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/transformers/models/auto/configuration_auto.py", line 1082, in from_pretrained
    config_dict, unused_kwargs = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/transformers/configuration_utils.py", line 644, in get_config_dict
    config_dict, kwargs = cls._get_config_dict(pretrained_model_name_or_path, **kwargs)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/transformers/configuration_utils.py", line 732, in _get_config_dict
    raise EnvironmentError(
OSError: It looks like the config file at '/data/models/tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf' is not a valid JSON file.

Code that was run

>>> from txtai.pipeline import LLM
>>> llm = LLM('/data/models/tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf', method='llama.cpp')

Directory with models

root@efe095bdf9d9:/app# ls -la /data/models
total 14103508
drwxr-xr-x 7 1000 1000       4096 Jan 13 00:51 .
drwxr-xr-x 3 root root       4096 Jan 13 00:51 ..
-rw-r--r-- 1 1000 1000  667814368 Jan 12 18:47 tinyllama-1.1b-1t-openorca.Q4_K_M.gguf
-rw-r--r-- 1 1000 1000  668788096 Jan 12 23:40 tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf

Do not understand the error, why the path is interpreted as a json config file... Don't you have thoughts about it?

davidmezzetti commented 10 months ago

Are you sure you're using txtai 6.3.0? This looks like an error that would happen with an older version.

4l1fe commented 10 months ago

Are you sure you're using txtai 6.3.0? This looks like an error that would happen with an older version.

Well, i installed 6.2.0. This version i have on my laptop while error i got by building docker container, so 6.2.0 should be fine. I guess i forgot the optional dependency txtai[pipeline] and fixing it now.

It brought me to a new complicated problem to find all the system dependencies for llama-cpp-python to build a docker image.

If you could advice something, please take a loot at the pip building traceback

requiremets.txt

...
txtai[pipeline]==6.3.0
...

Dockerfile

FROM python:3.11.7-slim-bookworm
ARG BUILD_DIR
WORKDIR /app
COPY $BUILD_DIR/requirements.txt /app
RUN pip install --no-cache-dir -r requirements.txt
COPY src /app

pip traceback while building an image

#0 99.51 Using cached wrapt-1.16.0-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (80 kB)
    #0 102.1 Building wheels for collected packages: fasttext, llama-cpp-python, tika
    #0 102.1   Building wheel for fasttext (setup.py): started
    #0 102.6   Building wheel for fasttext (setup.py): finished with status 'error'
    #0 102.6   error: subprocess-exited-with-error
    #0 102.6
    #0 102.6   × python setup.py bdist_wheel did not run successfully.
    #0 102.6   │ exit code: 1
    #0 102.6   ╰─> [66 lines of output]
    #0 102.6       /usr/local/lib/python3.11/site-packages/setuptools/dist.py:771: UserWarning: Usage of dash-separated 'description-file' will not be supported in future versions. Please use the underscore name 'description_file' instead
    #0 102.6         warnings.warn(
    #0 102.6       running bdist_wheel
    #0 102.6       running build
    #0 102.6       running build_py
    #0 102.6       creating build
    #0 102.6       creating build/lib.linux-x86_64-cpython-311
    #0 102.6       creating build/lib.linux-x86_64-cpython-311/fasttext
    #0 102.6       copying python/fasttext_module/fasttext/__init__.py -> build/lib.linux-x86_64-cpython-311/fasttext
    #0 102.6       copying python/fasttext_module/fasttext/FastText.py -> build/lib.linux-x86_64-cpython-311/fasttext
    #0 102.6       creating build/lib.linux-x86_64-cpython-311/fasttext/util
    #0 102.6       copying python/fasttext_module/fasttext/util/util.py -> build/lib.linux-x86_64-cpython-311/fasttext/util
    #0 102.6       copying python/fasttext_module/fasttext/util/__init__.py -> build/lib.linux-x86_64-cpython-311/fasttext/util
    #0 102.6       creating build/lib.linux-x86_64-cpython-311/fasttext/tests
    #0 102.6       copying python/fasttext_module/fasttext/tests/test_configurations.py -> build/lib.linux-x86_64-cpython-311/fasttext/tests
    #0 102.6       copying python/fasttext_module/fasttext/tests/__init__.py -> build/lib.linux-x86_64-cpython-311/fasttext/tests
    #0 102.6       copying python/fasttext_module/fasttext/tests/test_script.py -> build/lib.linux-x86_64-cpython-311/fasttext/tests
    #0 102.6       running build_ext
    #0 102.6       creating tmp
    #0 102.6       gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/usr/local/include/python3.11 -c /tmp/tmphfk85je_.cpp -o tmp/tmphfk85je_.o -std=c++14
    #0 102.6       gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/usr/local/include/python3.11 -c /tmp/tmphh7y0ixz.cpp -o tmp/tmphh7y0ixz.o -std=c++11
    #0 102.6       Traceback (most recent call last):
    #0 102.6         File "<string>", line 2, in <module>
    #0 102.6         File "<pip-setuptools-caller>", line 34, in <module>
    #0 102.6         File "/tmp/pip-install-d808p4t0/fasttext_cbb4d01961b64513ad8eb0e3e36c9abc/setup.py", line 168, in <module>
    #0 102.6           setup(
    #0 102.6         File "/usr/local/lib/python3.11/site-packages/setuptools/__init__.py", line 87, in setup
    #0 102.6           return distutils.core.setup(**attrs)
    #0 102.6                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    #0 102.6         File "/usr/local/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 185, in setup
    #0 102.6           return run_commands(dist)
    #0 102.6                  ^^^^^^^^^^^^^^^^^^
    #0 102.6         File "/usr/local/lib/python3.11/site-packages/setuptools/_distutils/core.py", line 201, in run_commands
    #0 102.6           dist.run_commands()
    #0 102.6         File "/usr/local/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 968, in run_commands
    #0 102.6           self.run_command(cmd)
    #0 102.6         File "/usr/local/lib/python3.11/site-packages/setuptools/dist.py", line 1217, in run_command
    #0 102.6           super().run_command(command)
    #0 102.6         File "/usr/local/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
    #0 102.6           cmd_obj.run()
    #0 102.6         File "/usr/local/lib/python3.11/site-packages/wheel/bdist_wheel.py", line 368, in run
    #0 102.6           self.run_command("build")
    #0 102.6         File "/usr/local/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 319, in run_command
    #0 102.6           self.distribution.run_command(command)
    #0 102.6         File "/usr/local/lib/python3.11/site-packages/setuptools/dist.py", line 1217, in run_command
    #0 102.6           super().run_command(command)
    #0 102.6         File "/usr/local/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
    #0 102.6           cmd_obj.run()
    #0 102.6         File "/usr/local/lib/python3.11/site-packages/setuptools/_distutils/command/build.py", line 132, in run
    #0 102.6           self.run_command(cmd_name)
    #0 102.6         File "/usr/local/lib/python3.11/site-packages/setuptools/_distutils/cmd.py", line 319, in run_command
    #0 102.6           self.distribution.run_command(command)
    #0 102.6         File "/usr/local/lib/python3.11/site-packages/setuptools/dist.py", line 1217, in run_command
    #0 102.6           super().run_command(command)
    #0 102.6         File "/usr/local/lib/python3.11/site-packages/setuptools/_distutils/dist.py", line 987, in run_command
    #0 102.6           cmd_obj.run()
    #0 102.6         File "/usr/local/lib/python3.11/site-packages/setuptools/command/build_ext.py", line 84, in run
    #0 102.6           _build_ext.run(self)
    #0 102.6         File "/usr/local/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py", line 346, in run
    #0 102.6           self.build_extensions()
    #0 102.6         File "/tmp/pip-install-d808p4t0/fasttext_cbb4d01961b64513ad8eb0e3e36c9abc/setup.py", line 146, in build_extensions
    #0 102.6           opts.append(cpp_flag(self.compiler))
    #0 102.6                       ^^^^^^^^^^^^^^^^^^^^^^^
    #0 102.6         File "/tmp/pip-install-d808p4t0/fasttext_cbb4d01961b64513ad8eb0e3e36c9abc/setup.py", line 108, in cpp_flag
    #0 102.6           raise RuntimeError(
    #0 102.6       RuntimeError: Unsupported compiler -- at least C++11 support is needed!
    #0 102.6       [end of output]
    #0 102.6
    #0 102.6   note: This error originates from a subprocess, and is likely not a problem with pip.
    #0 102.6   ERROR: Failed building wheel for fasttext
    #0 102.6   Running setup.py clean for fasttext
    #0 102.9   Building wheel for llama-cpp-python (pyproject.toml): started
    #0 103.5   Building wheel for llama-cpp-python (pyproject.toml): finished with status 'error'
    #0 103.5   error: subprocess-exited-with-error
    #0 103.5
    #0 103.5   × Building wheel for llama-cpp-python (pyproject.toml) did not run successfully.
    #0 103.5   │ exit code: 1
    #0 103.5   ╰─> [24 lines of output]
    #0 103.5       *** scikit-build-core 0.7.1 using CMake 3.28.1 (wheel)
    #0 103.5       *** Configuring CMake...
    #0 103.5       loading initial cache file /tmp/tmp9xeeh8dr/build/CMakeInit.txt
    #0 103.5       -- The C compiler identification is unknown
    #0 103.5       -- The CXX compiler identification is unknown
    #0 103.5       CMake Error at CMakeLists.txt:3 (project):
    #0 103.5         No CMAKE_C_COMPILER could be found.
    #0 103.5
    #0 103.5         Tell CMake where to find the compiler by setting either the environment
    #0 103.5         variable "CC" or the CMake cache entry CMAKE_C_COMPILER to the full path to
    #0 103.5         the compiler, or to the compiler name if it is in the PATH.
    #0 103.5
    #0 103.5
    #0 103.5       CMake Error at CMakeLists.txt:3 (project):
    #0 103.5         No CMAKE_CXX_COMPILER could be found.
    #0 103.5
    #0 103.5         Tell CMake where to find the compiler by setting either the environment
    #0 103.5         variable "CXX" or the CMake cache entry CMAKE_CXX_COMPILER to the full path
    #0 103.5         to the compiler, or to the compiler name if it is in the PATH.
    #0 103.5
    #0 103.5
    #0 103.5       -- Configuring incomplete, errors occurred!
    #0 103.5
    #0 103.5       *** CMake configuration failed
    #0 103.5       [end of output]
    #0 103.5
    #0 103.5   note: This error originates from a subprocess, and is likely not a problem with pip.
    #0 103.5   ERROR: Failed building wheel for llama-cpp-python
    #0 103.5   Building wheel for tika (setup.py): started
    #0 104.0   Building wheel for tika (setup.py): finished with status 'done'
    #0 104.0   Created wheel for tika: filename=tika-2.6.0-py3-none-any.whl size=32622 sha256=29f437f0069c8112a6ee9b2af1feda918cbb8bdef097ad27db85e2c25d680b89
    #0 104.0   Stored in directory: /root/.cache/pip/wheels/27/ba/2f/37420d1191bdae5e855d69b8e913673045bfd395cbd78ad697
    #0 104.0 Successfully built tika
    #0 104.0 Failed to build fasttext llama-cpp-python
    #0 104.0 ERROR: Could not build wheels for fasttext, llama-cpp-python, which is required to install pyproject.toml-based projects
    #0 104.0
    #0 104.0 [notice] A new release of pip is available: 23.2.1 -> 23.3.2
    #0 104.0 [notice] To update, run: pip install --upgrade pip

Or, hopefully, you know where to get prepared llama-cpp Dockerfile? I'm inspecting the following, but not sure that i'm on the right way.

4l1fe commented 10 months ago

Oh, i see something promising https://hub.docker.com/r/neuml/txtai-cpu 🙂

4l1fe commented 10 months ago

Oh, i see something promising https://hub.docker.com/r/neuml/txtai-cpu 🙂

Yeah, i lacked the system dependencies in the Dockerfile by link.

Done!

davidmezzetti commented 10 months ago

Glad you were able to get the issue resolved!