search

unslothai / unsloth

Finetune Llama 3.2, Mistral, Phi, Qwen 2.5 & Gemma LLMs 2-5x faster with 80% less memory
https://unsloth.ai
Apache License 2.0
18.26k stars 1.27k forks source link

Loading unsloth/mistral-7b-instruct-v0.2-bnb-4bit error #348

Closed WillsonAmalrajA closed 7 months ago

WillsonAmalrajA commented 7 months ago

I am trying to load mistral-7b-instruct-v0.2-bnb-4bit model from unsloth using the following

model, tokenizer = FastLanguageModel.from_pretrained( model_name = "models/unsloth/mistral-7b-instruct-v0.2-bnb-4bit", max_seq_length = max_seq_length, dtype = None, load_in_4bit = True, trust_remote_code=True, device_map='auto', )

I have downloaded the model files correctly. But I always end up with this error

==((====))== Unsloth: Fast Mistral patching release 2024.4 \ /| GPU: Tesla T4. Max memory: 14.578 GB. Platform = Linux. O^O/ _/ \ Pytorch: 2.2.2+cu121. CUDA = 7.5. CUDA Toolkit = 12.1. \ / Bfloat16 = FALSE. Xformers = 0.0.25.post1. FA = False. "-____-" Free Apache license: http://github.com/unslothai/unsloth Unused kwargs: ['quant_method']. These kwargs are not used in <class 'transformers.utils.quantization_config.BitsAndBytesConfig'>. Traceback (most recent call last): File "/home/jupyter/employee-experience-ai-ml/fine-tuning/unsloth_finetune.py", line 22, in model, tokenizer = FastLanguageModel.from_pretrained( File "/home/jupyter/employee-experience-ai-ml/fine-tuning/unsloth/lib/python3.10/site-packages/unsloth/models/loader.py", line 138, in from_pretrained model, tokenizer = dispatch_model.from_pretrained( File "/home/jupyter/employee-experience-ai-ml/fine-tuning/unsloth/lib/python3.10/site-packages/unsloth/models/mistral.py", line 361, in from_pretrained model = AutoModelForCausalLM.from_pretrained( File "/home/jupyter/employee-experience-ai-ml/fine-tuning/unsloth/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 563, in from_pretrained return model_class.from_pretrained( File "/home/jupyter/employee-experience-ai-ml/fine-tuning/unsloth/lib/python3.10/site-packages/transformers/modeling_utils.py", line 3456, in from_pretrained with safe_open(resolved_archive_file, framework="pt") as f: safetensors_rust.SafetensorError: Error while deserializing header: MetadataIncompleteBuffer

Never mind. It was model download error. Once I downloaded the model correctly I was able to move past this step

sanidhyasamaiya commented 6 months ago

Could you please elaborate how exactly you downloaded the model correctly? I am facing the same issue. I am having trouble loading an old model that i had finetuned, however the new model I finetuned loads without hiccups.