unslothai / unsloth

Finetune Llama 3.2, Mistral, Phi & Gemma LLMs 2-5x faster with 80% less memory
https://unsloth.ai
Apache License 2.0
17.7k stars 1.23k forks source link

Unable to load unsloth trained model saved to a local directory. #934

Open InderjeetVishnoi opened 2 months ago

InderjeetVishnoi commented 2 months ago

I created a tar file out of a unsloth fine-tuned model(base-model: unsloth/gemma-2b-bnb-4bit) using PEFT and pushed it to gcsBucket. I am downloading the artifacts from gcs bucket, extracting the files and locally saving them into a folder. Now, I am trying to load the model using this local directory path, but getting error : Exception: expected value at line 1 column 1

 model, tokenizer = FastLanguageModel.from_pretrained(
    model_name ='./model',
    max_seq_length = max_seq_length,
    dtype = None,
    load_in_4bit = True,
    local_files_only = True,
  )
 FastLanguageModel.for_inference(model)

How can I load the model from the given directory?

danielhanchen commented 2 months ago

Ok that's a weird issue - do you know if normal HF loading works?

InderjeetVishnoi commented 2 months ago

Yes, HF loading works fine. The key observation is : even loading from local works but its not working when a tar file is created from the model artifacts and is extracted back before loading.

danielhanchen commented 2 months ago

Ok will try investigating this - sorry on the issue!

InderjeetVishnoi commented 1 month ago

Hi Do we have any progress or update on this? Or if you could provide some workaround for meanwhile?

danielhanchen commented 1 month ago

Sorry been a bit busy apologies! Weird it should normally work since I detect a local inventory - how about passing just "model" and not "./model"

InderjeetVishnoi commented 1 month ago

I have tried that too , does not work. In-fact , recently encountered the same issue even without creating tar and extracting it. Steps are as below :

InderjeetVishnoi commented 4 weeks ago

Hi @danielhanchen Do we have any workaround for this, I am kind of blocked because of this issue. The moment model artifacts are placed form point A to point B, loading it from B doesn't works 🥲