AttributeError: 'Llama' object has no attribute 'ctx'

[h4rk8s]

Step 1 : download-model

llm llama-cpp download-model \
  https://huggingface.co/TheBloke/WizardCoder-15B-1.0-GGML/resolve/main/WizardCoder-15B-1.0.ggmlv3.q4_0.bin \
  --alias wizardcoder-15b --alias wc-15

Step2 : prompt it

llm -m wc-15 'give me a bash code, echo hello'

Error happend.

Error:
Exception ignored in: <function Llama.__del__ at 0x105e0ce00>
Traceback (most recent call last):
  File "/opt/homebrew/Cellar/llm/0.6.1/libexec/lib/python3.11/site-packages/llama_cpp/llama.py", line 1510, in __del__
    if self.ctx is not None:
       ^^^^^^^^
AttributeError: 'Llama' object has no attribute 'ctx'

[keijik]

I'm finding same as well

[cpilsworth]

I had a similar problem when my model file had not fully downloaded. Perhaps check the file size.

I've not tried the wizardcoder model, but I notice it says:

Please note that these GGMLs are not compatible with llama.cpp

on https://huggingface.co/TheBloke/WizardCoder-15B-1.0-GGML

[alexmill]

I've encountered this issue as well – I've found downloading llama.cpp repo and running the following command works on my M2 Ultra:

make -j && ./main -t 20 -ngl 40 -m "$(llm llama-cpp models-dir)/llama-2-70b-chat.ggmlv3.q5_K_M.bin" \ 
    -p "Building a website can be done in 10 simple steps:" --color -c 2048 --temp 0.7 --repeat_penalty 1.1 --no-mmap --ignore-eos -n 64 -gqa 8

I've pieced together from elsewhere the issue with these quantized models is related to the -gqa 8 flag.

When I run:

llm -m llama-2-70b-chat.ggmlv3.q5_K_M "hello world"

I just get Error: in response.