Unable to load unsloth trained model saved to a local directory.

[InderjeetVishnoi]

I created a tar file out of a unsloth fine-tuned model(base-model: unsloth/gemma-2b-bnb-4bit) using PEFT and pushed it to gcsBucket. I am downloading the artifacts from gcs bucket, extracting the files and locally saving them into a folder. Now, I am trying to load the model using this local directory path, but getting error : Exception: expected value at line 1 column 1

 model, tokenizer = FastLanguageModel.from_pretrained(
    model_name ='./model',
    max_seq_length = max_seq_length,
    dtype = None,
    load_in_4bit = True,
    local_files_only = True,
  )
 FastLanguageModel.for_inference(model)

How can I load the model from the given directory?

[danielhanchen]

Ok that's a weird issue - do you know if normal HF loading works?

[InderjeetVishnoi]

Yes, HF loading works fine. The key observation is : even loading from local works but its not working when a tar file is created from the model artifacts and is extracted back before loading.

[danielhanchen]

Ok will try investigating this - sorry on the issue!

[InderjeetVishnoi]

Hi Do we have any progress or update on this? Or if you could provide some workaround for meanwhile?

[danielhanchen]

Sorry been a bit busy apologies! Weird it should normally work since I detect a local inventory - how about passing just "model" and not "./model"

[InderjeetVishnoi]

I have tried that too , does not work. In-fact , recently encountered the same issue even without creating tar and extracting it. Steps are as below :

• Clone a fine-tuned model from HF
• upload it to a gcs bucket
• download it from bucket to a local directory
• specify the directory while loading the model.

[InderjeetVishnoi]

Hi @danielhanchen Do we have any workaround for this, I am kind of blocked because of this issue. The moment model artifacts are placed form point A to point B, loading it from B doesn't works 🥲