Improve Common Crawl download - Githubissues

NVIDIA / NeMo-Curator

Scalable toolkit for data curation

Apache License 2.0

327 stars 32 forks source link

Improve Common Crawl download #82

Closed ryantwolf closed 13 hours ago

ryantwolf commented 1 month ago

Description

Improves the download_common_crawl utility by adding more error handling and throwing more descriptive errors. Closes #80 .

Usage

Usage is the same as before.

Checklist

[x] I am familiar with the Contributing Guide.
[x] New or Existing tests cover these changes.
[x] The documentation is up to date with these changes.