Data for BERT, OpenAI GPT-2, BigBird (MT-DNN), ERNIE

szha commented 5 years ago

Hi all. Let's use this issue to discuss approach and challenges for reproducing the large transformer-based models such as GPT-2 (#592), BigBird (MT-DNN) (#633), ERNIE, as well as exploring their limits. We can also use this thread to track and pool resources that people find.

Data

The biggest challenge in reproducing GPT-2 is in curating the data. There are some efforts in the open source community (e.g. this one) though it's unclear how far these efforts have gone.
There are good corpora out there such as STORIES #598, resources in WMT2019.

hankcs commented 5 years ago

ERNIE team has released the preprocessed task corpora there, though they didn't release their corpora for pre-training. Maybe Chinese Gigaword is a good substitute?

szha commented 5 years ago

Here's a cleaned-up corpus from Trinh et. al., consisting of stories https://console.cloud.google.com/storage/browser/commonsense-reasoning/reproduce/stories_corpus?pli=1 https://arxiv.org/abs/1806.02847

hankcs commented 5 years ago

Here are several large scale raw corpora for Chinese: https://github.com/brightmart/nlp_chinese_corpus

dmlc / gluon-nlp

Data for BERT, OpenAI GPT-2, BigBird (MT-DNN), ERNIE #638

Data