Closed brand17 closed 3 years ago
See also same question on stackoverflow and the news data release announcement:
aws --no-sign-request s3 ls --recursive s3://commoncrawl/crawl-data/CC-NEWS/
or for a subset:
aws --no-sign-request s3 ls --recursive s3://commoncrawl/crawl-data/CC-NEWS/2017/09/
Thanks
I can obtain listing for Common crawl by:
https://commoncrawl.s3.amazonaws.com/crawl-data/CC-MAIN-2017-09/wet.paths.gz
How can I do this with Common Crawl News Dataset ?