search

deeppavlov / DeepPavlov

An open source library for deep learning end-to-end dialog systems and chatbots.
https://deeppavlov.ai
Apache License 2.0
6.71k stars 1.15k forks source link

Plans to trained bigger BERT/RoBERTa/T5 models for Russian language? #1139

Closed avostryakov closed 4 years ago

avostryakov commented 4 years ago

Hello!

Do you have any plans for training larger transformers models, something from the last architectures (Reformer specifically) or BERT?

Or maybe you have plans in the opposite direction: to train TinyBERT with https://github.com/huawei-noah/Pretrained-Language-Model/tree/master/TinyBERT ?

yurakuratov commented 4 years ago

Hi!

I can't say any specific plans on what is going to be released, but we are interested in training larger and more recent architectures for Russian language. I expect that to the end of this year or sooner we will release new models.

We also have ongoing work on models distillation but I can't name any time frames.

avostryakov commented 4 years ago

Thanks, Yura!