Not initialized model weights when loading checkpoint

[jacopobandonib]

When loading the model I get that some model weight are not initialized, is this expected?

from alignscore import AlignScore
scorer = AlignScore(model='roberta-large', batch_size=1, ckpt_path ="models/AlignScore/checkpoint/AlignScore-large.ckpt", device= "cpu", evaluation_mode='nli')

Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.dense.weight', 'lm_head.dense.bias', 'lm_head.bias', 'lm_head.layer_norm.weight', 'lm_head.layer_norm.bias']
- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of RobertaModel were not initialized from the model checkpoint at roberta-large and are newly initialized: ['roberta.pooler.dense.weight', 'roberta.pooler.dense.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Lightning automatically upgraded your loaded checkpoint from v1.8.0.post1 to v1.9.5. To apply the upgrade to your files permanently, run `python -m pytorch_lightning.utilities.upgrade_checkpoint --file [c:\Users\GOLYI\OneDrive](file:///C:/Users/GOLYI/OneDrive) - Bayer\Desktop\fact verification\FactFinder\eval_framework\models\AlignScore\checkpoint\AlignScore-large.ckpt`
Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.dense.weight', 'lm_head.dense.bias', 'lm_head.bias', 'lm_head.layer_norm.weight', 'lm_head.layer_norm.bias']
- This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of RobertaModel were not initialized from the model checkpoint at roberta-large and are newly initialized: ['roberta.pooler.dense.weight', 'roberta.pooler.dense.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

I am currently using Python 3.10.10, with torch==1.12.1

[supergirl-os]

I have the same question. Did you solve this?

[yuh-zha]

I think it's OK to ignore the warning message since we've changed the default structure of RoBERTa in Huggingface. It should work correctly.