search

lyogavin / airllm

AirLLM 70B inference with single 4GB GPU
Apache License 2.0
4.63k stars 370 forks source link

Issue `model.safetensors.index.json should exist` with loading model in safetensors format #185

Open LeMoussel opened 3 weeks ago

LeMoussel commented 3 weeks ago

When loading the model unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit, I have the error : AssertionError: model.safetensors.index.json should exist.

from airllm import AutoModel

model = AutoModel.from_pretrained("unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit")

Indeed in this model there is no file model.safetensors.index.json The same goes for the model unsloth/Meta-Llama-3.1-70B-bnb-4bit The file model.safetensors.index.json is present in the model unsloth/Meta-Llama-3.1-405B-Instruct-bnb-4bit

zeynepgulhanuslu commented 1 week ago

I get the same error. Could you fix this?