lyogavin / airllm

AirLLM 70B inference with single 4GB GPU
Apache License 2.0
5.34k stars 431 forks source link

Issue `model.safetensors.index.json should exist` with loading model in safetensors format #185

Open LeMoussel opened 2 months ago

LeMoussel commented 2 months ago

When loading the model unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit, I have the error : AssertionError: model.safetensors.index.json should exist.

from airllm import AutoModel

model = AutoModel.from_pretrained("unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit")

Indeed in this model there is no file model.safetensors.index.json The same goes for the model unsloth/Meta-Llama-3.1-70B-bnb-4bit The file model.safetensors.index.json is present in the model unsloth/Meta-Llama-3.1-405B-Instruct-bnb-4bit

zeynepgulhanuslu commented 1 month ago

I get the same error. Could you fix this?

werruww commented 1 month ago

I get the same error

karanbhatia007 commented 2 weeks ago

Same error