Unable to load mT5 with MT5Model.from_pretrained("google/mt5-small")

[Xuanfang1121]

Environment info

• transformers version: 4.2.2
• Platform: win10
• Python version: 3.6
• PyTorch version (GPU?): pytorch==1.8.1
• Tensorflow version (GPU?):
• Using GPU in script?:
• Using distributed or parallel set-up in script?:

Who can help

@LysandreJik@patrickvonplaten, @patil-suraj

Information

Model I am using (mt5): code

from transformers import MT5Model from transformers import T5Tokenizer

tokenizer = T5Tokenizer.from_pretrained("google/mt5-small") model = MT5Model.from_pretrained("google/mt5-small")

When I use the above code, the following error appears： Some weights of the model checkpoint at google/mt5-small were not used when initializing MT5Model: ['lm_head.weight']

• This IS expected if you are initializing MT5Model from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
• This IS NOT expected if you are initializing MT5Model from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).

https://huggingface.co/docs/transformers/model_doc/mt5

How to load the mt5-base(or mt5-small) model correctly? thanks

[patil-suraj]

Hi @Xuanfang1121 !

You are loading MT5Model , which loads the base model without the lm_head, so when loading the pre-trained weights the lm_head.weight is ignored.

If you want to load the model for conditional generation training or inference then you should use the MT5ForConditionalGeneration class which has the lm_head.

[Xuanfang1121]

Hi @Xuanfang1121 !

You are loading MT5Model , which loads the base model without the lm_head, so when loading the pre-trained weights the lm_head.weight is ignored.

If you want to load the model for conditional generation training or inference then you should use the MT5ForConditionalGeneration class which has the lm_head.

ok,thanks