[FIXED] RuntimeError: Unsloth: Your repo has a LoRA adapter and a base model.

[djannot]

I've trained the unsloth/Llama-3.2-3B-Instruct-bnb-4bit model successfully, but when I try to use it with astLanguageModel.from_pretrained, I get this error:

Traceback (most recent call last):
  File "/home/denis/Documents/ai/kubegen/server.py.gpu.unsloth.chat_template", line 17, in <module>
    model, tokenizer = FastLanguageModel.from_pretrained(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/denis/miniconda3/envs/unsloth_env/lib/python3.11/site-packages/unsloth/models/loader.py", line 196, in from_pretrained
    raise RuntimeError(
RuntimeError: Unsloth: Your repo has a LoRA adapter and a base model.
You have 2 files `config.json` and `adapter_config.json`.
We must only allow one config file.
Please separate the LoRA and base models to 2 repos.

It's weird because there's no config.json file:

-rw-rw-r-- 1 denis denis      738 Sep 26 09:28 adapter_config.json
-rw-rw-r-- 1 denis denis 97307544 Sep 26 09:28 adapter_model.safetensors
-rw-rw-r-- 1 denis denis     5112 Sep 26 09:28 README.md
-rw-rw-r-- 1 denis denis      454 Sep 26 09:28 special_tokens_map.json
-rw-rw-r-- 1 denis denis    51752 Sep 26 09:28 tokenizer_config.json
-rw-rw-r-- 1 denis denis 17209920 Sep 26 09:28 tokenizer.json

[aksh-n]

+1. I'm certain this issue was caused by something committed in the last 24 hours. I went back to the last commit on Sep 23 fb77505f8429566f5d21d6ea5318c342e8a67991, and there is no error.

[ibadrehman90]

Facing the same issue. The model was working fine and suddenly I started getting this error. The only change I did was to uninstall and re-installed transformers library.

[kholiavko-roman]

I am having the same issue.

[danielhanchen]

Sorry guys - does it work fine now? Please also reinstall and update Unsloth:

pip uninstall unsloth -y
pip install --upgrade --no-cache-dir "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"

[kholiavko-roman]

I started a fresh instance on colab collab and I got the same issue:

here is LoRA adapter that I am trying to load:

[djannot]

Sorry guys - does it work fine now? Please also reinstall and update Unsloth:

pip uninstall unsloth -y
pip install --upgrade --no-cache-dir "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"

Same issue after doing that

[aksh-n]

The issue might have to do something with transformers version. The issue surfaces when I use transformers version 4.45.0 (which was released yesterday), but not when I use 4.44.2 (the default version colab currently uses, and the one before 4.45.0). maybe the new versions of unsloth lead us to install the new version of transformers, causing the issue?

[danielhanchen]

Ok will investigate asap - sorry for the issue!

[vietphuon]

ran into the same issue, any suggestions?

[vietphuon]

So if I'm loading a LoRA adapter from the HF repo, this line should throw an exception and yield is_model = False right? I think next to investigate why this version of transformers's AutoConfig ran.

[danielhanchen]

@vietphuon Do you know what transformers version? 4.45? 4.46?

[danielhanchen]

Yes it seems like its a transformers issue - temporarily please use transformers<4.45 via pip install --upgrade --no-cache-dir "transformers<4.45.0" - I'll try seeing what the issue is - sorry guys!

[vietphuon]

https://github.com/huggingface/transformers/blob/main/src/transformers/models/auto/configuration_auto.py This thing commited 2 days ago, probably it. The timing lines up. https://github.com/huggingface/transformers/commit/f2c388e3f946862f657acc1e21b272ec946fc66c

[vietphuon]

@vietphuon Do you know what transformers version? 4.45? 4.46?

4.45.1

[djannot]

@danielhanchen when I try with 4.44.2 I get the following error:

Traceback (most recent call last):
  File "/home/denis/Documents/ai/kubegen/server.py.gpu.unsloth.chat_template", line 17, in <module>
    model, tokenizer = FastLanguageModel.from_pretrained(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/denis/miniconda3/envs/unsloth_env/lib/python3.11/site-packages/unsloth/models/loader.py", line 304, in from_pretrained
    model, tokenizer = dispatch_model.from_pretrained(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/denis/miniconda3/envs/unsloth_env/lib/python3.11/site-packages/unsloth/models/llama.py", line 1604, in from_pretrained
    tokenizer = load_correct_tokenizer(
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/denis/miniconda3/envs/unsloth_env/lib/python3.11/site-packages/unsloth/tokenizer_utils.py", line 538, in load_correct_tokenizer
    tokenizer = _load_correct_tokenizer(
                ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/denis/miniconda3/envs/unsloth_env/lib/python3.11/site-packages/unsloth/tokenizer_utils.py", line 496, in _load_correct_tokenizer
    fast_tokenizer = AutoTokenizer.from_pretrained(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/denis/miniconda3/envs/unsloth_env/lib/python3.11/site-packages/transformers/models/auto/tokenization_auto.py", line 897, in from_pretrained
    return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/denis/miniconda3/envs/unsloth_env/lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 2271, in from_pretrained
    return cls._from_pretrained(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/denis/miniconda3/envs/unsloth_env/lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 2505, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/denis/miniconda3/envs/unsloth_env/lib/python3.11/site-packages/transformers/tokenization_utils_fast.py", line 115, in __init__
    fast_tokenizer = TokenizerFast.from_file(fast_tokenizer_file)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Exception: data did not match any variant of untagged enum ModelWrapper at line 1251003 column 3

[kholiavko-roman]

@danielhanchen when I try with 4.44.2 I get the following error:

Traceback (most recent call last):
  File "/home/denis/Documents/ai/kubegen/server.py.gpu.unsloth.chat_template", line 17, in <module>
    model, tokenizer = FastLanguageModel.from_pretrained(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/denis/miniconda3/envs/unsloth_env/lib/python3.11/site-packages/unsloth/models/loader.py", line 304, in from_pretrained
    model, tokenizer = dispatch_model.from_pretrained(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/denis/miniconda3/envs/unsloth_env/lib/python3.11/site-packages/unsloth/models/llama.py", line 1604, in from_pretrained
    tokenizer = load_correct_tokenizer(
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/denis/miniconda3/envs/unsloth_env/lib/python3.11/site-packages/unsloth/tokenizer_utils.py", line 538, in load_correct_tokenizer
    tokenizer = _load_correct_tokenizer(
                ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/denis/miniconda3/envs/unsloth_env/lib/python3.11/site-packages/unsloth/tokenizer_utils.py", line 496, in _load_correct_tokenizer
    fast_tokenizer = AutoTokenizer.from_pretrained(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/denis/miniconda3/envs/unsloth_env/lib/python3.11/site-packages/transformers/models/auto/tokenization_auto.py", line 897, in from_pretrained
    return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/denis/miniconda3/envs/unsloth_env/lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 2271, in from_pretrained
    return cls._from_pretrained(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/denis/miniconda3/envs/unsloth_env/lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 2505, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/denis/miniconda3/envs/unsloth_env/lib/python3.11/site-packages/transformers/tokenization_utils_fast.py", line 115, in __init__
    fast_tokenizer = TokenizerFast.from_file(fast_tokenizer_file)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Exception: data did not match any variant of untagged enum ModelWrapper at line 1251003 column 3

Same error with pip install --upgrade --no-cache-dir "transformers<4.45.0"

[vietphuon]

@danielhanchen when I try with 4.44.2 I get the following error:

Traceback (most recent call last):
  File "/home/denis/Documents/ai/kubegen/server.py.gpu.unsloth.chat_template", line 17, in <module>
    model, tokenizer = FastLanguageModel.from_pretrained(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/denis/miniconda3/envs/unsloth_env/lib/python3.11/site-packages/unsloth/models/loader.py", line 304, in from_pretrained
    model, tokenizer = dispatch_model.from_pretrained(
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/denis/miniconda3/envs/unsloth_env/lib/python3.11/site-packages/unsloth/models/llama.py", line 1604, in from_pretrained
    tokenizer = load_correct_tokenizer(
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/denis/miniconda3/envs/unsloth_env/lib/python3.11/site-packages/unsloth/tokenizer_utils.py", line 538, in load_correct_tokenizer
    tokenizer = _load_correct_tokenizer(
                ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/denis/miniconda3/envs/unsloth_env/lib/python3.11/site-packages/unsloth/tokenizer_utils.py", line 496, in _load_correct_tokenizer
    fast_tokenizer = AutoTokenizer.from_pretrained(
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/denis/miniconda3/envs/unsloth_env/lib/python3.11/site-packages/transformers/models/auto/tokenization_auto.py", line 897, in from_pretrained
    return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/denis/miniconda3/envs/unsloth_env/lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 2271, in from_pretrained
    return cls._from_pretrained(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/denis/miniconda3/envs/unsloth_env/lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 2505, in _from_pretrained
    tokenizer = cls(*init_inputs, **init_kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/denis/miniconda3/envs/unsloth_env/lib/python3.11/site-packages/transformers/tokenization_utils_fast.py", line 115, in __init__
    fast_tokenizer = TokenizerFast.from_file(fast_tokenizer_file)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Exception: data did not match any variant of untagged enum ModelWrapper at line 1251003 column 3

I try @danielhanchen transformers==4.44.2 and it's working. I use kaggle with Llama 3.2 1b (load LoRA adapter from a HF repo)

[kiranpedvak]

im getting the same issue want to inference llam3.2 1B model , nothing is working to resolve the issue

the following errors:

when checked with new versions:

    RuntimeError: Unsloth: Your repo has a LoRA adapter and a base model.
    You have 2 files `config.json` and `adapter_config.json`.
    We must only allow one config file.
    Please separate the LoRA and base models to 2 repos.

when downgrade the transformers:

Exception: data did not match any variant of untagged enum ModelWrapper at line 1251003 column 3

[djannot]

@vietphuon I didn't have any issue with the 1B. It's only with the 3B.

@danielhanchen should I train the 3B model again after downgrading to 4.44.2 ?

[razzaghi]

I getting the same issue want to inference llam3.2 3B model , nothing is working to resolve the issue the following errors: RuntimeError: Unsloth: Your repo has a LoRA adapter and a base model. You have 2 files config.json and adapter_config.json. We must only allow one config file. Please separate the LoRA and base models to 2 repos. when downgrade the transformers: Exception: data did not match any variant of untagged enum ModelWrapper at line 1251003 column 3

[BenBatsir]

Any update? Tried downgrading transformers but it didn't do the trick. I wonder if I can simply comment out the RuntimeError on loader.py

[danielhanchen]

Apologies everyone!! Will get to the bottom of this today - thank you so much for your patience and sorry on the delay!!

[danielhanchen]

Much much apologies just found the issue! It seems like transformers 4.45 is now using a weird way to detect if a LoRA was trained or not. I fixed it now! Please update Unsloth on local machines via:

No need to retrain LoRAs! Also do NOT downgrade transformers - use 4.45.1 or higher

pip uninstall unsloth -y && pip install --upgrade --no-cache-dir "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"

For Colab and Kaggle, no need to update - just restart and delete runtime.

Sorry on the issue again! @razzaghi @djannot @ibadrehman90 @kholiavko-roman @BenBatsir @aksh-n @vietphuon @kiranpedvak

@vietphuon I also tried vietphuon/Llama-3.2-1B-Instruct-bnb-4bit-240926-1 - it should function now as well!

Apologies everyone and thank you for your patience!

[djannot]

It works like a charm !

Thanks a lot @danielhanchen for your reactivity

[paulokuriki]

Thank you so much, @danielhanchen

[vietphuon]

Thanks, it's working now @danielhanchen

[rinabuoy]

The current version leads the below exception, when loading a LORA model: Exception: data did not match any variant of untagged enum ModelWrapper at line 1251003 column 3

[danielhanchen]

@rinabuoy You'll need to downgrade transformers temporarily to 4.44.2 and then you'll need to re-upload the tokenizer again

[cici-tan]

for version unsloth==2024.10.3 and transformers==4.44.2, still get the same issue: Exception: data did not match any variant of untagged enum ModelWrapper at line 1250944 column 3