low_level_api_chat_cpp.py - ModuleNotFoundError: No module named 'common'

[lucasjinreal]

Hello, does a chat-cli example avaiable?

[abetlen]

@lucasjinreal is this what you're looking for https://github.com/abetlen/llama-cpp-python/blob/main/examples/low_level_api/low_level_api_chat_cpp.py ?

[lucasjinreal]

@abetlen it looks like this script can not be run with latest lama.cpp

[gjmulder]

$ LLAMA_CUBLAS=1 pip install --force-reinstall --ignore-installed --no-cache-dir llama-cpp-python
Collecting llama-cpp-python
  Downloading llama_cpp_python-0.1.49.tar.gz (1.1 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 29.0 MB/s eta 0:00:00
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting typing-extensions>=4.5.0
  Downloading typing_extensions-4.5.0-py3-none-any.whl (27 kB)
Building wheels for collected packages: llama-cpp-python
  Building wheel for llama-cpp-python (pyproject.toml) ... done
  Created wheel for llama-cpp-python: filename=llama_cpp_python-0.1.49-cp310-cp310-linux_x86_64.whl size=196284 sha256=e80a7a6aaf5eadcb78190b148b6166b003213bdd566603f15979e313108ba24c
  Stored in directory: /data/tmp/pip-ephem-wheel-cache-sht5p4l9/wheels/01/f3/f7/b070cbde244babc2ead31326c399bd2d4187ffffed9ea09ce9
Successfully built llama-cpp-python
Installing collected packages: typing-extensions, llama-cpp-python
Successfully installed llama-cpp-python-0.1.49 typing-extensions-4.5.0

$ python3 ./low_level_api_chat_cpp.py 
Traceback (most recent call last):
  File "/var/tmp/./low_level_api_chat_cpp.py", line 19, in <module>
    from common import GptParams, gpt_params_parse, gpt_random_prompt
ModuleNotFoundError: No module named 'common'

[enzolutions]

Hi, here are my two cents to try to find a solution.

Do you need to copy from https://github.com/abetlen/llama-cpp-python/tree/main/examples/low_level_api the following files:

• utils.py
• low_level_api_chat_cpp.py
• common.py

In addition, I update a little bit the low_level_api_chat_cpp.py

change llama_cpp.llama_eval -> llama_cpp.llama_decode Add an extra parameter FALSE to llama_cpp.llama_token

However, I'm encountering difficulties with the llama_decode function. Specifically, the batch parameter requires significant re-work. Regrettably, my understanding of this library is insufficient to resolve the issue.