[Numpy Refactor] Model wishlist in GluonNLP

[sxjscience]

There are a lots of recent backbone models for NLP task. In our initial attempt of the new version of GluonNLP, we added these models:

• BERT
• ALBERT
• ELECTRA
• RoBERTa
• XLMR
• MobileBERT
• Transformer
• TransformerXL

We will keep adding models and the following are some backbone (model) architectures that are good to have

• [ ] T5 PyTorch: https://huggingface.co/transformers/model_doc/t5.html TF: https://github.com/google-research/text-to-text-transfer-transformer
• [x] BART PyTorch: https://github.com/pytorch/fairseq/blob/master/fairseq/models/bart/model.py
• [ ] Longformer PyTorch: https://github.com/allenai/longformer
• [ ] Sparse Transformer We may use the Jukebox implementation: https://github.com/openai/jukebox/blob/master/jukebox/transformer/factored_attention.py

[sxjscience]

Need to add also

• [ ] VAMPIRE: https://github.com/allenai/vampire
• [ ] Funnel Transformer: https://arxiv.org/pdf/2006.03236.pdf

[szha]

there are some new ELECTRA pre-trained models in Chinese that seem useful: https://github.com/ymcui/HFL-Anthology#hfl-anthology