Open jowagner opened 2 years ago
OSCAR 21.09 uses a newer version of CommonCrawl. Changelog does not mention any improvements in the language filter. Paper: https://hal.inria.fr/hal-03301590
Issues with OSCAR corpora for low-resource languages: https://arxiv.org/abs/2103.12028 / https://hal.inria.fr/hal-03177623
Q: Does this use the new pipeline or the old one?
Related: issue #77
OSCAR 21.09 uses a newer version of CommonCrawl. Changelog does not mention any improvements in the language filter. Paper: https://hal.inria.fr/hal-03301590
Issues with OSCAR corpora for low-resource languages: https://arxiv.org/abs/2103.12028 / https://hal.inria.fr/hal-03177623
Q: Does this use the new pipeline or the old one?
Related: issue #77