search

huawei-noah / Pretrained-Language-Model

Pretrained language model and its related optimization techniques developed by Huawei Noah's Ark Lab.
3.02k stars 628 forks source link

Including task loss #139

Open shairoz-deci opened 3 years ago

shairoz-deci commented 3 years ago

Thank you for sharing this repo. I have a question about the loss used to train TinyBert. Unlike DistilBert, MobileBert and other distillation based BERT variants, TinyBert training doesn't include the the student loss w.r.t the actual label but only to that of the teacher, not even in the task specific training. Have you tried including it in the loss function?

Thanks in advance,

zwjyyc commented 3 years ago

Hi, we have tried this loss, but we have not seen improvement.