Open szha opened 5 years ago
ERNIE team has released the preprocessed task corpora there, though they didn't release their corpora for pre-training. Maybe Chinese Gigaword is a good substitute?
Here's a cleaned-up corpus from Trinh et. al., consisting of stories https://console.cloud.google.com/storage/browser/commonsense-reasoning/reproduce/stories_corpus?pli=1 https://arxiv.org/abs/1806.02847
Here are several large scale raw corpora for Chinese: https://github.com/brightmart/nlp_chinese_corpus
Hi all. Let's use this issue to discuss approach and challenges for reproducing the large transformer-based models such as GPT-2 (#592), BigBird (MT-DNN) (#633), ERNIE, as well as exploring their limits. We can also use this thread to track and pool resources that people find.
Data