search

arcee-ai / mergekit

Tools for merging pretrained large language models.
GNU Lesser General Public License v3.0
4.79k stars 434 forks source link

Merges Gemma models are not loading in llamacpp-python or LM studio with Gemma support #181

Closed rombodawg closed 7 months ago

rombodawg commented 8 months ago

Llamacpp-python (text generation web ui dev branch) and LM studio have both added support for Gemma models. However when merging Gemma models, then converting to GGUF, the resulting model does not load in either UI.

Lm studio error:

"llama.cpp error: 'create_tensor: tensor 'output.weight' not found'"

Text generation web ui dev branch (llamacpp-python) error

Traceback (most recent call last):

File "E:\Open_source_ai_chatbot\OOBA_11-Dev\text-generation-webui\modules\ui_model_menu.py", line 242, in load_model_wrapper

shared.model, shared.tokenizer = load_model(selected_model, loader)

                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\Open_source_ai_chatbot\OOBA_11-Dev\text-generation-webui\modules\models.py", line 87, in load_model

output = load_func_map[loader](model_name)

         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\Open_source_ai_chatbot\OOBA_11-Dev\text-generation-webui\modules\models.py", line 250, in llamacpp_loader

model, tokenizer = LlamaCppModel.from_pretrained(model_file)

                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "E:\Open_source_ai_chatbot\OOBA_11-Dev\text-generation-webui\modules\llamacpp_model.py", line 102, in from_pretrained

result.model = Llama(**params)

               ^^^^^^^^^^^^^^^
File "E:\Open_source_ai_chatbot\OOBA_11-Dev\text-generation-webui\installer_files\env\Lib\site-packages\llama_cpp_cuda\llama.py", line 313, in init

self._model = _LlamaModel(

              ^^^^^^^^^^^^
File "E:\Open_source_ai_chatbot\OOBA_11-Dev\text-generation-webui\installer_files\env\Lib\site-packages\llama_cpp_cuda_internals.py", line 55, in init

raise ValueError(f"Failed to load model from file: {path_model}")
ValueError: Failed to load model from file: models\model\Gemma-EveryoneLLM-7b-test-fp16.gguf
rombodawg commented 8 months ago

Sorry if the issue isnt originating with you. I honestly dont know who to open the issues with at this point. Ive already opens like 4 other issues, with llamacpp, oobagooba, and lm studio

Using llamacpp convert.py to quantize, oobagooba and lm studio to inference

rombodawg commented 8 months ago

Im uploading the model files for the merges if anyone wants to do some debugging. Should be in the next 10 hours or so. Sorry slow internet.

Follow the mulit-thread. And check out my model for debugging.

Thread links: https://github.com/lmstudio-ai/configs/issues/21 https://github.com/ggerganov/llama.cpp/issues/5706 https://github.com/arcee-ai/mergekit/issues/181 https://github.com/oobabooga/text-generation-webui/issues/5562

https://huggingface.co/rombodawg/Gemme-Merge-Test-7b

rombodawg commented 7 months ago

I just want to point out this issue has been resolved