huawei-noah / Pretrained-Language-Model

Pretrained language model and its related optimization techniques developed by Huawei Noah's Ark Lab.
3.03k stars 627 forks source link

Orca support and 4 new models #243

Closed ghaddarAbs closed 10 months ago

ghaddarAbs commented 10 months ago

Add support for tasks from ORCA leaderboard, as well as adding support for 4 new models: JABERv2, JABERv2-6L, AT5Bv2, AT5Sv2. The update are to support a new arxiv paper titled On the importance of Data Scale in Pretraining Arabic Language Models that will appear online in a couple of days.