ERROR text-generation-webui???

[kangz543g]

Describe the bug

text-generation-webui???

Is there an existing issue for this?

• [X] I have searched the existing issues

Reproduction

09:47:30-526026 INFO Loading "upstage_SOLAR-10.7B-Instruct-v1.0" 09:47:30-526026 ERROR Failed to load the model. Traceback (most recent call last): File "C:\text-generation-webui\modules\ui_model_menu.py", line 245, in load_model_wrapper shared.model, shared.tokenizer = load_model(selected_model, loader) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\text-generation-webui\modules\models.py", line 87, in load_model output = load_func_maploader ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\text-generation-webui\modules\models.py", line 247, in llamacpp_loader model_file = list(Path(f'{shared.args.model_dir}/{model_name}').glob('*.gguf'))[0]

IndexError: list index out of range  

### Screenshot

_No response_

### Logs

```shell
text-generation-webui???
```

### System Info

```shell
text-generation-webui???
```

[Galaxia-mk]

Your model seems to be in the wrong format. Try to load this model which seems to be the same model but in the format that you can run if you have enough vram. https://huggingface.co/TheBloke/SOLAR-10.7B-Instruct-v1.0-GGUF/tree/main

Just download the the files called: gitattributes
config.json

and which ever .gguf file you like into the same map. They range from "solar-10.7b-instruct-v1.0.Q2_K.gguf" to "solar-10.7b-instruct-v1.0.Q8_0.gguf". Where Q2 takes up the least amount of VRAM and Q8 the most. The higher the number after the Q which stands for quantization the higher the performance.

Then put that map into the models folder of oobabooga and run the model using the .ccp loader. And if you want to exclusively use the GPU then put the slider called "n-gpu-layers" all the way to the right. I hope this helped

[mtx-z]

@Galaxia-mk thank you for this explanation. How do you manage multiple gguf models ? As I think downloading another guff model override the config.json file. It doesn't seem that subfolders are supported. Thank you.

[Galaxia-mk]

@Galaxia-mk thank you for this explanation. How do you manage multiple gguf models ? As I think downloading another guff model override the config.json file. It doesn't seem that subfolders are supported. Thank you.

I put every gguf file I download in a separate folder together with the gitattributes and config.json files of that model. When I make a new map for a new model I usually just copy and paste its name which in this case would be "solar-10.7b-instruct-v1.0.Q8_0.gguf" so I can tell what it is exactly later easily.

Sorry for the late reply, I barely log in on this website.

[github-actions[bot]]

This issue has been closed due to inactivity for 6 months. If you believe it is still relevant, please leave a comment below. You can tag a developer in your comment.