search

oobabooga / text-generation-webui

A Gradio web UI for Large Language Models.
GNU Affero General Public License v3.0
38.63k stars 5.1k forks source link

suppport for ggmlv3 coming on llama.cpp? #2245

Closed yesbroc closed 1 year ago

yesbroc commented 1 year ago

so far it seems like it hasn't been implemented even through a reinstall. i am using TheBloke's wizard-mega 13b ggmlv3

yesbroc commented 1 year ago 
The following models are available:

1. New folder
2. wizard-mega-13B.ggmlv3.q5_1.bin
3. wizard-vicuna-13B.ggmlv3.q4_1.bin
4. WizardLM-7B-uncensored.ggmlv3.q4_1.bin

Which one do you want to load? 1-4

2

INFO:Loading wizard-mega-13B.ggml.q5_1.bin...
INFO:llama.cpp weights detected: models\wizard-mega-13B.ggml.q5_1.bin

INFO:Cache capacity is 8000 bytes
llama.cpp: loading model from models\wizard-mega-13B.ggml.q5_1.bin
error loading model: unknown (magic, version) combination: 67676a74, 00000003; is this really a GGML file?
llama_init_from_file: failed to load model
Traceback (most recent call last):
  File "C:\Users\orijp\OneDrive\Desktop\chatgpts\oobabooga_windows\oobabooga_windows\text-generation-webui\server.py", line 1033, in <module>
    shared.model, shared.tokenizer = load_model(shared.model_name)
  File "C:\Users\orijp\OneDrive\Desktop\chatgpts\oobabooga_windows\oobabooga_windows\text-generation-webui\modules\models.py", line 95, in load_model
    output = load_func(model_name)
  File "C:\Users\orijp\OneDrive\Desktop\chatgpts\oobabooga_windows\oobabooga_windows\text-generation-webui\modules\models.py", line 258, in llamacpp_loader
    model, tokenizer = LlamaCppModel.from_pretrained(model_file)
  File "C:\Users\orijp\OneDrive\Desktop\chatgpts\oobabooga_windows\oobabooga_windows\text-generation-webui\modules\llamacpp_model.py", line 50, in from_pretrained
    self.model = Llama(**params)
  File "C:\Users\orijp\OneDrive\Desktop\chatgpts\oobabooga_windows\oobabooga_windows\installer_files\env\lib\site-packages\llama_cpp\llama.py", line 159, in __init__
    assert self.ctx is not None
AssertionError
Exception ignored in: <function LlamaCppModel.__del__ at 0x0000013CC099D480>
Traceback (most recent call last):
  File "C:\Users\orijp\OneDrive\Desktop\chatgpts\oobabooga_windows\oobabooga_windows\text-generation-webui\modules\llamacpp_model.py", line 23, in __del__
    self.model.__del__()
AttributeError: 'LlamaCppModel' object has no attribute 'model'

Done!
Press any key to continue . . .
m-spangenberg commented 1 year ago

I'm sure they'll bump llama-cpp-python the moment this PR gets merged: https://github.com/abetlen/llama-cpp-python/pull/251