The dataset is part of the High Performance Language Technologies project (HPLT), a 3-year EU-funded project started in September 2022. HPLT derives monolingual and bilingual datasets from the Internet Archive and CommonCrawl and builds efficient and solid machine translation (MT) as well as large language models (LLMs). HPLT aims at providing free, sustainable and reusable datasets, models and workflows at scale using high-performance computing (HPC).
Dataloader name:
hplt/hplt.py
DataCatalogue: http://seacrowd.github.io/seacrowd-catalogue/card.html?hplt