Closed stefan-it closed 3 years ago
Decided it would be easier for us to take care of this since we plan to directly release the model checkpoint in huggingface.
Started working on it over the week-end, will share PR once it is more polished.
This is great news @Iwontbecreative! Let us know if you need help.
🌟 New model addition
Model description
Hi,
I just found this really interesting upcoming ICLR 2021 paper: "Rethinking Embedding Coupling in Pre-trained Language Models":
Paper can be found here.
Thus, the authors propose a new Rebalanced mBERT (RemBERT) model that outperforms XLM-R. An integration into Transformers would be awesome!
I would really like to help with the integration into Transformers, as soon as the model is out!
Open source status