AttributeError: 'LlamaCppModel' object has no attribute 'model'

[Vadimluck]

Describe the bug

Installed "text-generation-webui" and "vicuna-13b-cocktail" worked, but some others didn't want to work and I decided to (reinstall) uninstall and install from scratch "text-generation-webui". Now "vicuna-13b-cocktail" stopped working. An error appears in the console:

Which one do you want to load? 1-2

2

INFO:Loading vicuna-13b-cocktail...
INFO:llama.cpp weights detected: models\vicuna-13b-cocktail\ggml-vicuna-13b-cocktail-v1-q8_0.bin

INFO:Cache capacity is 0 bytes
llama.cpp: loading model from models\vicuna-13b-cocktail\ggml-vicuna-13b-cocktail-v1-q8_0.bin
Traceback (most recent call last):
  File "C:\chatGPT\text-generation-webui\server.py", line 1087, in <module>
    shared.model, shared.tokenizer = load_model(shared.model_name)
  File "C:\chatGPT\text-generation-webui\modules\models.py", line 95, in load_model
    output = load_func(model_name)
  File "C:\chatGPT\text-generation-webui\modules\models.py", line 272, in llamacpp_loader
    model, tokenizer = LlamaCppModel.from_pretrained(model_file)
  File "C:\chatGPT\text-generation-webui\modules\llamacpp_model.py", line 50, in from_pretrained
    self.model = Llama(**params)
  File "C:\chatGPT\installer_files\env\lib\site-packages\llama_cpp\llama.py", line 158, in __init__
    self.ctx = llama_cpp.llama_init_from_file(
  File "C:\chatGPT\installer_files\env\lib\site-packages\llama_cpp\llama_cpp.py", line 262, in llama_init_from_file
    return _lib.llama_init_from_file(path_model, params)
OSError: [WinError -529697949] Windows Error 0xe06d7363
Exception ignored in: <function Llama.__del__ at 0x000001E7EC5D32E0>
Traceback (most recent call last):
  File "C:\chatGPT\installer_files\env\lib\site-packages\llama_cpp\llama.py", line 1236, in __del__
    if self.ctx is not None:
AttributeError: 'Llama' object has no attribute 'ctx'
Exception ignored in: <function LlamaCppModel.__del__ at 0x000001E7EC5D2560>
Traceback (most recent call last):
  File "C:\chatGPT\text-generation-webui\modules\llamacpp_model.py", line 23, in __del__
    self.model.__del__()
AttributeError: 'LlamaCppModel' object has no attribute 'model'

Done!

Is there an existing issue for this?

• [X] I have searched the existing issues

Reproduction

1. Installed "text-generation-webui".
2. Unpacked on the way: "\text-generation-webui\models" in the folder "vicuna-13b-cocktail"

Screenshot

Logs

INFO:Loading vicuna-13b-cocktail...
INFO:llama.cpp weights detected: models\vicuna-13b-cocktail\ggml-vicuna-13b-cocktail-v1-q8_0.bin

INFO:Cache capacity is 0 bytes
llama.cpp: loading model from models\vicuna-13b-cocktail\ggml-vicuna-13b-cocktail-v1-q8_0.bin
Traceback (most recent call last):
  File "C:\chatGPT\text-generation-webui\server.py", line 1087, in <module>
    shared.model, shared.tokenizer = load_model(shared.model_name)
  File "C:\chatGPT\text-generation-webui\modules\models.py", line 95, in load_model
    output = load_func(model_name)
  File "C:\chatGPT\text-generation-webui\modules\models.py", line 272, in llamacpp_loader
    model, tokenizer = LlamaCppModel.from_pretrained(model_file)
  File "C:\chatGPT\text-generation-webui\modules\llamacpp_model.py", line 50, in from_pretrained
    self.model = Llama(**params)
  File "C:\chatGPT\installer_files\env\lib\site-packages\llama_cpp\llama.py", line 158, in __init__
    self.ctx = llama_cpp.llama_init_from_file(
  File "C:\chatGPT\installer_files\env\lib\site-packages\llama_cpp\llama_cpp.py", line 262, in llama_init_from_file
    return _lib.llama_init_from_file(path_model, params)
OSError: [WinError -529697949] Windows Error 0xe06d7363
Exception ignored in: <function Llama.__del__ at 0x000001D4489472E0>
Traceback (most recent call last):
  File "C:\chatGPT\installer_files\env\lib\site-packages\llama_cpp\llama.py", line 1236, in __del__
    if self.ctx is not None:
AttributeError: 'Llama' object has no attribute 'ctx'
Exception ignored in: <function LlamaCppModel.__del__ at 0x000001D448946560>
Traceback (most recent call last):
  File "C:\chatGPT\text-generation-webui\modules\llamacpp_model.py", line 23, in __del__
    self.model.__del__()
AttributeError: 'LlamaCppModel' object has no attribute 'model'

Done!

System Info

Device name Fox
Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz 2.21 GHz
32.0 GB RAM (available: 31.9 GB)
Device code 7E29F2FB-C2B7-44F5-8D82-718D4519F01A
Product code 00325-81363-09802-AAOEM
System type 64-bit operating system, x64 processor
Pen and touch input Pen and touch input are not available for this monitor
Windows 11 Home Edition
Version 21H2
Installation date 06.10.2022
OS build 22000.978
Windows Feature Experience Pack 1000.22000.978.0 Interaction
Model: Predator PT515-51

[Undone]

The model is using an older version of GGML, there was an update and only GGMLv3 works now. #2264

[Vadimluck]

Thank you very much! My problem was solved as soon as I downloaded q5, before that I was using q8.

[araleza]

More recently (March 2024), I've been seeing this error come up again. This time, it's due to being out of VRAM. Try sliding the n-gpu-layers slider down to 1 and reloading, and if that works, then see how far back up you can put it without the error message reappearing.

I think that something has started using more VRAM, as I can't set the n-gpu-layers value as high as I think I used to.

[agentmith]

More recently (March 2024), I've been seeing this error come up again. This time, it's due to being out of VRAM. Try sliding the n-gpu-layers slider down to 1 and reloading, and if that works, then see how far back up you can put it without the error message reappearing.

I think that something has started using more VRAM, as I can't set the n-gpu-layers value as high as I think I used to.

Having this issue with CohereForAI/c4ai-command-r-v01-4bit models on a 3090, with quantizations ~20 gig file size. I can load other models which are larger fine, and setting the GPU layers to 1 still results in this error. Not sure how to proceed.

[rmusser01]

Just received this error in attempting to load the IQ4XS quant from https://huggingface.co/MaziyarPanahi/WizardLM-2-8x22B-GGUF/tree/main I have 3 P40s running concurrently, I am using the latest release of text-gen-webui using the built-in update script.

When sliding the 'n-gpu-layers' to 1, the model loads successfully. Set to max layers, it fails. Set to 40 layers, it fails. Set to 30 layers, it succeeds Set to 20 layers, it succeeds.

Error log:

Traceback (most recent call last):
  File "/home/ml-user-1/LLMs/Runners/text-generation-webui/modules/ui_model_menu.py", line 249, in load_model_wrapper
    shared.model, shared.tokenizer = load_model(selected_model, loader)
                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ml-user-1/LLMs/Runners/text-generation-webui/modules/models.py", line 94, in load_model
    output = load_func_map[loader](model_name)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ml-user-1/LLMs/Runners/text-generation-webui/modules/models.py", line 272, in llamacpp_loader
    model, tokenizer = LlamaCppModel.from_pretrained(model_file)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ml-user-1/LLMs/Runners/text-generation-webui/modules/llamacpp_model.py", line 103, in from_pretrained
    result.model = Llama(**params)
                   ^^^^^^^^^^^^^^^
  File "/home/ml-user-1/LLMs/Runners/text-generation-webui/installer_files/env/lib/python3.11/site-packages/llama_cpp_cuda/llama.py", line 352, in __init__
    self._ctx = _LlamaContext(
                ^^^^^^^^^^^^^^
  File "/home/ml-user-1/LLMs/Runners/text-generation-webui/installer_files/env/lib/python3.11/site-packages/llama_cpp_cuda/_internals.py", line 267, in __init__
    raise ValueError("Failed to create llama_context")
ValueError: Failed to create llama_context

Exception ignored in: <function LlamaCppModel.__del__ at 0x7f5aa5b00b80>
Traceback (most recent call last):
  File "/home/ml-user-1/LLMs/Runners/text-generation-webui/modules/llamacpp_model.py", line 58, in __del__
    del self.model
        ^^^^^^^^^^
AttributeError: 'LlamaCppModel' object has no attribute 'model'

System info:

NAME="Debian GNU/Linux"
VERSION="12 (bookworm)"
Linux ML-Host-1 6.1.0-21-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.90-1 (2024-05-03) x86_64 GNU/Linux
model name      : Intel(R) Xeon(R) CPU E5-2680 v4 @ 2.40GHz
192 GB ram

nvidia-smi

|   0  Tesla P40                      On  |   00000000:03:00.0 Off |                  Off |
|   1  Tesla P40                      On  |   00000000:04:00.0 Off |                  Off |
|   2  Tesla P40                      On  |   00000000:A1:00.0 Off |                  Off |

[swizzcheeze]

lower your context length "n_ctx" try 4096

[adgorynov]

lower your context length "n_ctx" try 4096

Thank you, this advice has helped me!