How can I do MLM task on new dataset with new vocab.txt utilizing AlbertForPreTraining with pretrained weights?

lonePatient / albert_pytorch

A Lite Bert For Self-Supervised Learning Language Representations

Apache License 2.0

710 stars 152 forks source link

How can I do MLM task on new dataset with new vocab.txt utilizing AlbertForPreTraining with pretrained weights? #47

Open ChineseYjh opened 4 years ago

ChineseYjh commented 4 years ago

Hi~ I want to utilize AlbertForPreTraining to do MaskedLM task on new datasets(with new vocab.txt, whose size is not 21128) based on the pretraining weights. How can I do that precisely?

After I read the codes, I found decoder(Linear) in predictions(AlbertLMPredictionHead) in cls(AlbertPreTrainingHeads) in AlbertForPreTraining has output size of 21128, which seems hard to adapt to my new vocab size..

Thank you very much~