How to change type_vocab_size?

[leisurehippo]

tensorflow==2.3.1

transformers==4.2.1

My code is as follow:

transformer = TFAutoModel.from_pretrained('hfl/chinese-roberta-wwm-ext', from_pt=False, type_vocab_size=3)

I got error:

ValueError: cannot reshape array of size 1536 into shape (3,768)

[NielsRogge]

The embedding layer of the exisiting pretrained model you are loading has shape (2, 768). If you specify another shape, like (3, 768), then it cannot reshape the existing embedding layer to the new shape you are asking.

You can however circumvent this using the new ignore_mismatched_sizes argument:

from transformers import TFAutoModel

transformer = TFAutoModel.from_pretrained('hfl/chinese-roberta-wwm-ext', from_pt=False, type_vocab_size=3,
                                          ignore_mismatched_sizes=True)

This will print the following warning:

All model checkpoint layers were used when initializing TFBertModel.

Some weights of TFBertModel were not initialized from the model checkpoint at hfl/chinese-roberta-wwm-ext and are newly initialized because the shapes did not match:
- bert/embeddings/token_type_embeddings/embeddings:0: found shape (2, 768) in the checkpoint and (3, 768) in the model instantiated
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.