How large is the dataset

commoncrawl / news-crawl

News crawling with StormCrawler - stores content as WARC

Apache License 2.0

323 stars 35 forks source link

Closed sljlp closed 2 years ago

sljlp commented 2 years ago

Please tell me how large the dataset is. Thanks.

sljlp commented 2 years ago

What's the cleaning method?

sebastian-nagel commented 2 years ago

the Common Crawl news dataset from 2016 until Sep 2022 includes about 25 TiB of WARC files
no cleaning is done - the WARC files contain the HTML of the crawled news articles
for further information about the news dataset, see
- https://commoncrawl.org/2016/10/news-dataset-available/
- https://data.commoncrawl.org/crawl-data/CC-NEWS/index.html
for questions about Common Crawl datasets: https://groups.google.com/g/common-crawl