search

facebookresearch / fairseq

Facebook AI Research Sequence-to-Sequence Toolkit written in Python.
MIT License
30.43k stars 6.4k forks source link

Does fairseq support other pre-training language models like BERT, GPT, ELMO? #2282

Closed 14H034160212 closed 4 years ago

14H034160212 commented 4 years ago

❓ Questions and Help

Does fairseq support other pre-training language models like BERT, GPT, ELMO?

Thank you so much.

lematt1991 commented 4 years ago

Is there a particular model you're looking for? There are other language models, ex: roberta, xlmr, bart, mbart, etc.

14H034160212 commented 4 years ago

Is there a particular model you're looking for? There are other language models, ex: roberta, xlmr, bart, mbart, etc.

Yes, does any format required for the model? Like in the official example, the form of roberta model is like model.pt. So, if I want to use BERT, GPT or ELMO, shall I transfer the model firstly to the "model.pt" format and then run the script? Thank you so much.

myleott commented 4 years ago

Please take a look at the examples here: https://github.com/pytorch/fairseq/tree/master/examples

We don't have ELMo implemented, RoBERTa is similar to BERT, and you can train GPT-like models following the instructions here: https://github.com/pytorch/fairseq/tree/master/examples/language_model

14H034160212 commented 4 years ago

Please take a look at the examples here: https://github.com/pytorch/fairseq/tree/master/examples

We don't have ELMo implemented, RoBERTa is similar to BERT, and you can train GPT-like models following the instructions here: https://github.com/pytorch/fairseq/tree/master/examples/language_model

Thanks for your comment!