add TinyBERT?

[Dicksonchin93]

🌟 New model addition

Model description

TinyBERT is a smaller version of the Base BERT model, it uses transformer distillation (a type of knowledge distillation) to transfer the plenty of knowledge encoded in a large “teacher” BERT to a small “student” TinyBERT. is empirically effective and achieves more than 96% the performance of teacher BERTBASE on GLUE benchmark, while being 7.5x smaller and 9.4x faster on inference. TinyBERT is also significantly better than state-of-the-art baselines on BERT distillation, with only ∼28% parameters and ∼31% inference time of them. Here I have feature request to add the pretrained weights of TinyBERT after general learning from https://github.com/huawei-noah/Pretrained-Language-Model and model for both TF2.0 and pytorch. I think the transformer distillation method should be introduced too. https://arxiv.org/pdf/1909.10351.pdf

Open source status

• [x] the model implementation is available: (give details) https://github.com/huawei-noah/Pretrained-Language-Model, only pytorch is available in my knowledge at the moment https://github.com/koursaros-ai/nboost
• [x] the model weights are available: (give details) https://github.com/huawei-noah/Pretrained-Language-Model
• [x] who are the authors: (mention them, if possible by @gh-username) https://github.com/huawei-noah/Pretrained-Language-Model @jacobrxz https://github.com/jacobrxz

[huseinzol05]

tinybert just a normal bert with smaller parameters. I am not sure whether huggingface team will create new object called TinyBert. I think you can simply contact huawei-noah first to get permission to upload tinybert using your personal account.

[julien-c]

Or you could ask them if they would create an org account and upload TinyBert there.

I'll also ping them as it would be really great (cc @jacobrxz)

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.