`/models/available` broken for Huggingface gallery

[ape-nq]

LocalAI version:

OK:

• local-ai-avx2-Linux-x86_64-1.40.0
• local-ai-avx2-Linux-x86_64-2.0.0
• local-ai-avx2-Linux-x86_64-2.8.0
• local-ai-avx2-Linux-x86_64-2.8.2
• local-ai-avx2-Linux-x86_64-2.9.0

Broken:

• local-ai-cuda12-Linux-x86_64-2.10.1
• local-ai-cuda12-Linux-x86_64-2.11.0
• local-ai-cuda12-Linux-x86_64-2.12.0
• local-ai-cuda12-Linux-x86_64-2.12.4
• local-ai-avx2-Linux-x86_64-2.12.4
• local-ai-cuda12-Linux-x86_64-2.13.0

Environment, CPU architecture, OS, and Version:

$ uname -a
Linux 2d083df80d05 6.5.0-1018-oem #19-Ubuntu SMP PREEMPT_DYNAMIC Thu Mar 14 21:40:46 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Describe the bug

Cannot get Huggingface gallery in recent versions:

• GALLERIES='[{"url": "github:go-skynet/model-gallery/huggingface.yaml","name":"huggingface"}]'

"Index" model gallery works:

• GALLERIES='[{"name":"model-gallery", "url":"github:go-skynet/model-gallery/index.yaml"}]'

To Reproduce

1. Set Huggingface gallery:
   • export GALLERIES='[{"url": "github:go-skynet/model-gallery/huggingface.yaml","name":"huggingface"}]'
2. Start LocalAI binary
   • e.g. ./local-ai-avx2-Linux-x86_64-2.12.4
3. Try to view Gallery in Browser: http://localhost:8080/models/available
   • up to v2.9.0:
     • Log: 200 - GET /models/available
     • Browser: JSON result is shown
   • from v2.10.1 to v2.12.4:
     • Log: 500 - GET /models/available
     • Browser: {"error":{"code":500,"message":"json: unsupported type: map[interface {}]interface {}","type":""}}

Expected behavior

JSON for Huggingface gallery should be returned as in versions v2.9.0 and prior. Or as it is returned for standard 'model-gallery'.

Logs

Debug-Log is 15MB, so not attaching it here.

There is a really long line with what looks like a hex array:

• Starts with: 10:32AM DBG Models found from galleries: [0xc00032a9c0 0xc00032ad00 0xc00032add0 0xc00032b040
• Ends with: 0xc0090a2f70 0xc0090a3040 0xc0090a3110 0xc0090a31e0 0xc0090a32b0 0xc0090a3380 0xc0090a3450 0xc0090a3520 0xc0090a35f0 0xc0090a36c0 0xc0090a3790 0xc0090a3860 0xc0090a3930 0xc0090a3a00]

Excerpt with Model found messages:

10:32AM DBG Model found from galleries: &{URL:github:go-skynet/model-gallery/base.yaml Name:1tuanpham__instruct_en-vi_12000_1e_thebloke_mistralic-7b-1-gptq_lora_causal_lm__instruct_en-vi_12000_1e__adapter_model.bin Description: License: URLs:[https://huggingface.co/1TuanPham/Instruct_en-vi_12000_1e_TheBloke_Mistralic-7B-1-GPTQ_LORA_CAUSAL_LM] Icon: Tags:[peft region:us] ConfigFile:map[] Overrides:map[parameters:map[model:Instruct_en-vi_12000_1e/adapter_model.bin]] AdditionalFiles:[{Filename:Instruct_en-vi_12000_1e/adapter_model.bin SHA256: URI:https://huggingface.co/1TuanPham/Instruct_en-vi_12000_1e_TheBloke_Mistralic-7B-1-GPTQ_LORA_CAUSAL_LM/resolve/main/Instruct_en-vi_12000_1e/adapter_model.bin}] Gallery:{URL:github:go-skynet/model-gallery/huggingface.yaml Name:huggingface} Installed:false}
10:32AM DBG Model found from galleries: &{URL:github:go-skynet/model-gallery/base.yaml Name:1tuanpham__instruct_en-vi_14000_1e_b60_lr2e-4_thebloke_mistralic-7b-1-gptq_lora_causal_lm__adapter_model.bin Description: License: URLs:[https://huggingface.co/1TuanPham/Instruct_en-vi_14000_1e_b60_lr2e-4_TheBloke_Mistralic-7B-1-GPTQ_LORA_CAUSAL_LM] Icon: Tags:[peft region:us] ConfigFile:map[] Overrides:map[parameters:map[model:adapter_model.bin]] AdditionalFiles:[{Filename:adapter_model.bin SHA256: URI:https://huggingface.co/1TuanPham/Instruct_en-vi_14000_1e_b60_lr2e-4_TheBloke_Mistralic-7B-1-GPTQ_LORA_CAUSAL_LM/resolve/main/adapter_model.bin}] Gallery:{URL:github:go-skynet/model-gallery/huggingface.yaml Name:huggingface} Installed:false}
10:32AM DBG Model found from galleries: &{URL:github:go-skynet/model-gallery/base.yaml Name:1tuanpham__instruct_en-vi_14000_1e_b60_lr2e-4_thebloke_mistralic-7b-1-gptq_lora_causal_lm__instruct_en-vi_14000_1e_b60_lr2e-4__adapter_model.bin Description: License: URLs:[https://huggingface.co/1TuanPham/Instruct_en-vi_14000_1e_b60_lr2e-4_TheBloke_Mistralic-7B-1-GPTQ_LORA_CAUSAL_LM] Icon: Tags:[peft region:us] ConfigFile:map[] Overrides:map[parameters:map[model:Instruct_en-vi_14000_1e_b60_lr2e-4/adapter_model.bin]] AdditionalFiles:[{Filename:Instruct_en-vi_14000_1e_b60_lr2e-4/adapter_model.bin SHA256: URI:https://huggingface.co/1TuanPham/Instruct_en-vi_14000_1e_b60_lr2e-4_TheBloke_Mistralic-7B-1-GPTQ_LORA_CAUSAL_LM/resolve/main/Instruct_en-vi_14000_1e_b60_lr2e-4/adapter_model.bin}] Gallery:{URL:github:go-skynet/model-gallery/huggingface.yaml Name:huggingface} Installed:false}
10:32AM DBG Model found from galleries: &{URL:github:go-skynet/model-gallery/base.yaml Name:1tuanpham__instruct_en-vi_14500_1e_thebloke_mistralic-7b-1-gptq_lora_causal_lm__adapter_model.bin Description: License: URLs:[https://huggingface.co/1TuanPham/Instruct_en-vi_14500_1e_TheBloke_Mistralic-7B-1-GPTQ_LORA_CAUSAL_LM] Icon: Tags:[peft region:us] ConfigFile:map[] Overrides:map[parameters:map[model:adapter_model.bin]] AdditionalFiles:[{Filename:adapter_model.bin SHA256: URI:https://huggingface.co/1TuanPham/Instruct_en-vi_14500_1e_TheBloke_Mistralic-7B-1-GPTQ_LORA_CAUSAL_LM/resolve/main/adapter_model.bin}] Gallery:{URL:github:go-skynet/model-gallery/huggingface.yaml Name:huggingface} Installed:false}
10:32AM DBG Model found from galleries: &{URL:github:go-skynet/model-gallery/base.yaml Name:1tuanpham__instruct_en-vi_14500_1e_thebloke_mistralic-7b-1-gptq_lora_causal_lm__instruct_en-vi_14500_1e__adapter_model.bin Description: License: URLs:[https://huggingface.co/1TuanPham/Instruct_en-vi_14500_1e_TheBloke_Mistralic-7B-1-GPTQ_LORA_CAUSAL_LM] Icon: Tags:[peft region:us] ConfigFile:map[] Overrides:map[parameters:map[model:Instruct_en-vi_14500_1e/adapter_model.bin]] AdditionalFiles:[{Filename:Instruct_en-vi_14500_1e/adapter_model.bin SHA256: URI:https://huggingface.co/1TuanPham/Instruct_en-vi_14500_1e_TheBloke_Mistralic-7B-1-GPTQ_LORA_CAUSAL_LM/resolve/main/Instruct_en-vi_14500_1e/adapter_model.bin}] Gallery:{URL:github:go-skynet/model-gallery/huggingface.yaml Name:huggingface} Installed:false}
10:32AM DBG Model found from galleries: &{URL:github:go-skynet/model-gallery/base.yaml Name:1tuanpham__instruct_en-vi_7500_1epoch_thebloke_mistralic-7b-1-gptq_lora_causal_lm__adapter_model.bin Description: License: URLs:[https://huggingface.co/1TuanPham/Instruct_en-vi_7500_1epoch_TheBloke_Mistralic-7B-1-GPTQ_LORA_CAUSAL_LM] Icon: Tags:[peft region:us] ConfigFile:map[] Overrides:map[parameters:map[model:adapter_model.bin]] AdditionalFiles:[{Filename:adapter_model.bin SHA256: URI:https://huggingface.co/1TuanPham/Instruct_en-vi_7500_1epoch_TheBloke_Mistralic-7B-1-GPTQ_LORA_CAUSAL_LM/resolve/main/adapter_model.bin}] Gallery:{URL:github:go-skynet/model-gallery/huggingface.yaml Name:huggingface} Installed:false}
10:32AM DBG Model found from galleries: &{URL:github:go-skynet/model-gallery/base.yaml Name:1tuanpham__instruct_en-vi_7500_1epoch_thebloke_mistralic-7b-1-gptq_lora_causal_lm__instruct_en-vi_7500_1epoch__adapter_model.bin Description: License: URLs:[https://huggingface.co/1TuanPham/Instruct_en-vi_7500_1epoch_TheBloke_Mistralic-7B-1-GPTQ_LORA_CAUSAL_LM] Icon: Tags:[peft region:us] ConfigFile:map[] Overrides:map[parameters:map[model:Instruct_en-vi_7500_1epoch/adapter_model.bin]] AdditionalFiles:[{Filename:Instruct_en-vi_7500_1epoch/adapter_model.bin SHA256: URI:https://huggingface.co/1TuanPham/Instruct_en-vi_7500_1epoch_TheBloke_Mistralic-7B-1-GPTQ_LORA_CAUSAL_LM/resolve/main/Instruct_en-vi_7500_1epoch/adapter_model.bin}] Gallery:{URL:github:go-skynet/model-gallery/huggingface.yaml Name:huggingface} Installed:false}
10:32AM DBG Model found from galleries: &{URL:github:go-skynet/model-gallery/base.yaml Name:1tuanpham__instruct_en-vi_8000_1e_thebloke_mistralic-7b-1-gptq_lora_causal_lm__adapter_model.bin Description: License: URLs:[https://huggingface.co/1TuanPham/Instruct_en-vi_8000_1e_TheBloke_Mistralic-7B-1-GPTQ_LORA_CAUSAL_LM] Icon: Tags:[peft region:us] ConfigFile:map[] Overrides:map[parameters:map[model:adapter_model.bin]] AdditionalFiles:[{Filename:adapter_model.bin SHA256: URI:https://huggingface.co/1TuanPham/Instruct_en-vi_8000_1e_TheBloke_Mistralic-7B-1-GPTQ_LORA_CAUSAL_LM/resolve/main/adapter_model.bin}] Gallery:{URL:github:go-skynet/model-gallery/huggingface.yaml Name:huggingface} Installed:false}

Additional context

Running from release binaries on a rented VM. Binaries downloaded from GitHub and renamed to include version number.

[localai-bot]

It seems like the issue you're facing is related to a compatibility problem between the Hugging Face transformers library version and the models used. The error messages suggest that the models found in the galleries are not compatible with the version of transformers you are using.

The models mentioned are likely SF-T5 models, which require a specific version of the transformers library to work properly. You mentioned using transformers==4.6.0, but these models may require a different version. Specifically, the error suggests that you may need to use transformers==4.5.0.

To resolve this issue, you should update the transformers library in your environment to version 4.5.0. You can do this with pip by running the following command:

pip install transformers==4.5.0

After installing the correct version of the transformers library, try running your code again to see if the issue is resolved. If you continue to encounter problems, make sure that the models you are trying to load are compatible not only with the transformers library version but also with the Python version you are using.

[ape-nq]

This is not about running any model.

Just trying to list the gallery as described in https://localai.io/models/#list-models

Repro:

1. Start LocalAI
2. List gallery
3. Error

$ curl http://localhost:8080/models/available
{"error":{"code":500,"message":"json: unsupported type: map[interface {}]interface {}","type":""}}

[ape-nq]

Maybe the full debug log will help:

• localai-log-threads-.txt

The only request to LocalAI after startup is listing gallery models:

10:08AM INF core/startup process completed!
10:08AM DBG No configuration file found at /tmp/localai/upload/uploadedFiles.json
10:08AM DBG No configuration file found at /tmp/localai/config/assistants.json
10:08AM DBG No configuration file found at /tmp/localai/config/assistantsFile.json
10:08AM DBG Listing models from galleries: [{URL:github:go-skynet/model-gallery/huggingface.yaml Name:huggingface}]

[lenaxia]

I'm also able to replicate this, model-gallery works, hugging face does not.

[lopezl]

I'm also able to replicate this, model-gallery works, hugging face does not.

Another confirmation here. I can get a model list using only the main gallery but not using only huggingface gallery, using both at the same time also fails.

Currently running v2.12.4 (0004ec8be3ca150ce6d8b79f2991bfe3a9dc65ad) with the AIO docker image latest-aio-gpu-nvidia-cuda-11 a6ffa9444dd5

With DEBUG=true I can see that LocalAI reads and parses entire huggingface.yaml, last lines from the log matches with last lines from the YAML file), but finally fails to build the model list.

$ curl  $LOCAL_AI/models/available
{"error":{"code":500,"message":"json: unsupported type: map[interface {}]interface {}","type":""}}

Relevant log lines:

localai  | 8:00AM DBG Model found from galleries: &{URL:github:go-skynet/model-gallery/base.yaml Name:zhaijunxiao__pygmalion-13b-ggml__ggml-model-q4_0.bin Description: License: URLs:[https://huggingface.co/zhaijunxiao/Pygmalion-13B-ggml] Icon: Tags:[region:us] ConfigFile:map[] Overrides:map[parameters:map[model:ggml-model-q4_0.bin]] AdditionalFiles:[{Filename:ggml-model-q4_0.bin SHA256: URI:https://huggingface.co/zhaijunxiao/Pygmalion-13B-ggml/resolve/main/ggml-model-q4_0.bin}] Gallery:{URL:github:go-skynet/model-gallery/huggingface.yaml Name:model-gallery} Installed:false}
localai  | 8:00AM DBG Model found from galleries: &{URL:github:go-skynet/model-gallery/base.yaml Name:zinccat__santacoder-ggml-quantized__gpt_bigcode-santacoder-ggml-q4_1.bin Description: License:bigcode-openrail-m URLs:[https://huggingface.co/zinccat/santacoder-ggml-quantized] Icon: Tags:[license:bigcode-openrail-m region:us] ConfigFile:map[] Overrides:map[parameters:map[model:gpt_bigcode-santacoder-ggml-q4_1.bin]] AdditionalFiles:[{Filename:gpt_bigcode-santacoder-ggml-q4_1.bin SHA256: URI:https://huggingface.co/zinccat/santacoder-ggml-quantized/resolve/main/gpt_bigcode-santacoder-ggml-q4_1.bin}] Gallery:{URL:github:go-skynet/model-gallery/huggingface.yaml Name:model-gallery} Installed:false}

[devexa]

Same error on my side too with huggingface, using latest-aio-gpu-nvidia-cuda-12 docker image

[mKenfenheuer]

Same here

localai/localai:latest-aio-cpu sha256:a0924e805f9dc8cc5337e63442aca35463cdc10ab0f4b46fd2337a978974f8bf

[ape-nq]

Models in Huggingface gallery can be browsed using /browse/ endpoint in version 2.13.0.

The /models/available endpoint still throws errors in version 2.13.0:

$ curl http://localhost:8080/models/available
{"error":{"code":500,"message":"json: unsupported type: map[interface {}]interface {}","type":""}}

[ape-nq]

After checking out the new documentation, I noticed Huggingface gallery from go-skynet is not mentioned anymore.

Testing new recommended GALLERIES setting in version 2.13.0:

• using: export GALLERIES='[{"name":"localai", "url":"github:mudler/localai/gallery/index.yaml"}]'
• /browse/ is empty in browser
• /models/available endpoint throws new errors:
  • $ curl http://localhost:8080/models/available {"error":{"code":500,"message":"yaml: unmarshal errors:\n line 1: cannot unmarshal !!map into []*gallery.GalleryModel","type":""}}

When not setting GALLERIES:

• /browse/ shows models

• /models/available throws errors as for old Huggingface gallery:

  • $ curl http://localhost:8080/models/available {"error":{"code":500,"message":"json: unsupported type: map[interface {}]interface {}","type":""}}

  Not sure whether to track this in a new issue or continue this one.