missing weights

[neoyipeng2018]

Hi,

Thanks for releasing the pre-trained models on huggingface, have a basic question, was hoping you guys can help.

I loaded the MaskedLM model using the AutoModelForMaskedLM code.

from transformers import AutoTokenizer, AutoModelForMaskedLM tokenizer = AutoTokenizer.from_pretrained("hfl/chinese-macbert-base") model = AutoModelForMaskedLM.from_pretrained("hfl/chinese-macbert-base")

However, I received the following error, which I didn't expect, given the pre-trained weights were for maskedLM:

Some weights of the model checkpoint at hfl/chinese-macbert-base were not used when initializing BertForMaskedLM: ['cls.seq_relationship.weight', 'cls.seq_relationship.bias']

Wondering if you guys know why?

Thanks in advance, Yi Peng

[ymcui]

Greetings.

If you need to load the entire model, you should use BertForPreTraining. The BertForMaskedLM will ONLY load the weights for MLM (Mac in our paper), and thus it reports the missing of ['cls.seq_relationship.weight', 'cls.seq_relationship.bias'].

from transformers import BertModel, BertForPreTraining, BertForMaskedLM
>>> model = BertModel.from_pretrained("hfl/chinese-macbert-base")          # OK
>>> model = BertForPreTraining.from_pretrained("hfl/chinese-macbert-base") # OK
>>> model = transformers.BertForMaskedLM.from_pretrained("hfl/chinese-macbert-base")  # ONLY for MLM
Some weights of the model checkpoint at hfl/chinese-macbert-base were not used when initializing BertForMaskedLM: ['cls.seq_relationship.weight', 'cls.seq_relationship.bias']
- This IS expected if you are initializing BertForMaskedLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertForMaskedLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).

[neoyipeng2018]

Thanks so much for your quick reply!

On Fri, 12 Mar 2021, 08:38 Yiming Cui, @.***> wrote:

Greetings.

If you need to load the entire model, you should use BertForPreTraining. The BertForMaskedLM will ONLY load the weights for MLM (Mac in our paper), and thus it reports the missing of ['cls.seq_relationship.weight', 'cls.seq_relationship.bias'].

from transformers import BertModel, BertForPreTraining, BertForMaskedLM>>> model = BertModel.from_pretrained("hfl/chinese-macbert-base") # OK>>> model = BertForPreTraining.from_pretrained("hfl/chinese-macbert-base") # OK>>> model = transformers.BertForMaskedLM.from_pretrained("hfl/chinese-macbert-base") # ONLY for MLMSome weights of the model checkpoint at hfl/chinese-macbert-base were not used when initializing BertForMaskedLM: ['cls.seq_relationship.weight', 'cls.seq_relationship.bias']- This IS expected if you are initializing BertForMaskedLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).- This IS NOT expected if you are initializing BertForMaskedLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ymcui/MacBERT/issues/3#issuecomment-797151151, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKHA2K5TK36NO5CVMQJIINDTDFPADANCNFSM4ZBGXLCA .