TypeError: LlamaRotaryEmbedding.__init__() got an unexpected keyword argument 'config'

[DaddyCodesAlot]

Hi there, so I am loading a finetuned Llama 2 13b model, and I get this error.

Here's part of the error:

File /usr/local/lib/python3.10/dist-packages/unsloth/models/loader.py:172, in FastLanguageModel.from_pretrained(model_name, max_seq_length, dtype, load_in_4bit, token, device_map, rope_scaling, fix_tokenizer, trust_remote_code, use_gradient_checkpointing, resize_model_vocab, revision, *args, *kwargs) 169 tokenizer_name = None 170 pass --> 172 model, tokenizer = dispatch_model.from_pretrained( 173 model_name = model_name, 174 max_seq_length = max_seq_length, 175 dtype = dtype, 176 load_in_4bit = load_in_4bit, 177 token = token, 178 device_map = device_map, 179 rope_scaling = rope_scaling, 180 fix_tokenizer = fix_tokenizer, 181 model_patcher = dispatch_model, 182 tokenizer_name = tokenizer_name, 183 trust_remote_code = trust_remote_code, 184 revision = revision if not is_peft else None, 185 args, **kwargs, 186 ) 188 if resize_model_vocab is not None: 189 model.resize_token_embeddings(resize_model_vocab)

File /usr/local/lib/python3.10/dist-packages/unsloth/models/llama.py:1251, in FastLlamaModel.from_pretrained(model_name, max_seq_length, dtype, load_in_4bit, token, device_map, rope_scaling, fix_tokenizer, model_patcher, tokenizer_name, trust_remote_code, kwargs) 1249 max_position_embeddings = max(max_seq_length, model_max_seq_length) 1250 kwargs.pop("attn_implementation", None); # No need since we auto call it -> 1251 model = AutoModelForCausalLM.from_pretrained( 1252 model_name, 1253 device_map = device_map, 1254 torch_dtype = dtype, 1255 quantization_config = bnb_config, 1256 token = token, 1257 max_position_embeddings = max_position_embeddings, 1258 trust_remote_code = trust_remote_code, 1259 attn_implementation = "eager", 1260 kwargs, 1261 ) 1262 # Return old flag 1263 os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = old_hf_transfer

File /usr/local/lib/python3.10/dist-packages/transformers/models/auto/auto_factory.py:564, in _BaseAutoModelClass.from_pretrained(cls, pretrained_model_name_or_path, *model_args, *kwargs) 562 elif type(config) in cls._model_mapping.keys(): 563 model_class = _get_model_class(config, cls._model_mapping) --> 564 return model_class.from_pretrained( 565 pretrained_model_name_or_path, model_args, config=config, hub_kwargs, kwargs 566 ) 567 raise ValueError( 568 f"Unrecognized configuration class {config.class} for this kind of AutoModel: {cls.name}.\n" 569 f"Model type should be one of {', '.join(c.name for c in cls._model_mapping.keys())}." 570 )

File /usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py:3775, in PreTrainedModel.from_pretrained(cls, pretrained_model_name_or_path, config, cache_dir, ignore_mismatched_sizes, force_download, local_files_only, token, revision, use_safetensors, *model_args, *kwargs) 3769 config = cls._autoset_attn_implementation( 3770 config, use_flash_attention_2=use_flash_attention_2, torch_dtype=torch_dtype, device_map=device_map 3771 ) 3773 with ContextManagers(init_contexts): 3774 # Let's make sure we don't run the init function of buffer modules -> 3775 model = cls(config, model_args, **model_kwargs) 3777 # make sure we use the model's config since the init call might have copied it 3778 config = model.config

File /usr/local/lib/python3.10/dist-packages/transformers/models/llama/modeling_llama.py:1066, in LlamaForCausalLM.init(self, config) 1064 def init(self, config): 1065 super().init(config) -> 1066 self.model = LlamaModel(config) 1067 self.vocab_size = config.vocab_size 1068 self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)

File /usr/local/lib/python3.10/dist-packages/transformers/models/llama/modeling_llama.py:845, in LlamaModel.init(self, config) 841 self.vocab_size = config.vocab_size 843 self.embed_tokens = nn.Embedding(config.vocab_size, config.hidden_size, self.padding_idx) 844 self.layers = nn.ModuleList( --> 845 [LlamaDecoderLayer(config, layer_idx) for layer_idx in range(config.num_hidden_layers)] 846 ) 847 self.norm = LlamaRMSNorm(config.hidden_size, eps=config.rms_norm_eps) 848 self.rotary_emb = LlamaRotaryEmbedding(config=config)

File /usr/local/lib/python3.10/dist-packages/transformers/models/llama/modeling_llama.py:845, in (.0) 841 self.vocab_size = config.vocab_size 843 self.embed_tokens = nn.Embedding(config.vocab_size, config.hidden_size, self.padding_idx) 844 self.layers = nn.ModuleList( --> 845 [LlamaDecoderLayer(config, layer_idx) for layer_idx in range(config.num_hidden_layers)] 846 ) 847 self.norm = LlamaRMSNorm(config.hidden_size, eps=config.rms_norm_eps) 848 self.rotary_emb = LlamaRotaryEmbedding(config=config)

File /usr/local/lib/python3.10/dist-packages/transformers/models/llama/modeling_llama.py:632, in LlamaDecoderLayer.init(self, config, layer_idx) 629 super().init() 630 self.hidden_size = config.hidden_size --> 632 self.self_attn = LLAMA_ATTENTION_CLASSES[config._attn_implementation](config=config, layer_idx=layer_idx) 634 self.mlp = LlamaMLP(config) 635 self.input_layernorm = LlamaRMSNorm(config.hidden_size, eps=config.rms_norm_eps)

File /usr/local/lib/python3.10/dist-packages/transformers/models/llama/modeling_llama.py:306, in LlamaAttention.init(self, config, layer_idx) 303 self.o_proj = nn.Linear(self.hidden_size, self.hidden_size, bias=config.attention_bias) 305 # TODO (joao): remove in v4.45 (RoPE is computed in the model, not in the decoder layers) --> 306 self.rotary_emb = LlamaRotaryEmbedding(config=self.config)

TypeError: LlamaRotaryEmbedding.init() got an unexpected keyword argument 'config'

[DJAlexJ]

guess this commit in transformers is related: https://github.com/huggingface/transformers/commit/2e113422b3504fe6de821bb9911b24273b11aa9c

[DJAlexJ]

This also is being raised when trying to download new Llama-3.1, since there is a new type of transform: https://github.com/huggingface/transformers/commit/d5a99dfcee6e94065cb7c83cc8ab6fc5daa0cc4e

[DaddyCodesAlot]

This also is being raised when trying to download new Llama-3.1, since there is a new type of transform: huggingface/transformers@d5a99df

Yeah, I'll try rolling back transformers to a previous commit and see if it resolves the bug for Llama 2 models.

[DJAlexJ]

Wow, there is already a PR from @danielhanchen, good job

[DaddyCodesAlot]

Oh wow, these guys are soldiers! Thanks @danielhanchen!

[adityajkorade]

Has this issue been resolved? Tried using the 'Llama-3.1 8b + Unsloth 2x faster finetuning.ipynb' to download the 'unsloth/Meta-Llama-3.1-8B', it gave me same same issue, are there any fix you know of @DaddyCodesAlot ?

[danielhanchen]

Oh please reinstall and update Unsloth! I'll reopen this for people!

pip uninstall unsloth -y
pip install --upgrade --force-reinstall --no-cache-dir git+https://github.com/unslothai/unsloth.git

[leochang123]

I am still unable to resolve this issue: 'TypeError: LlamaRotaryEmbedding.init() got an unexpected keyword argument 'config'.' Any advice or suggestions would be greatly appreciated.

[danielhanchen]

@leochang123 Did you try uninstalling unsloth and reinstalling?