Collect samples of formats from Common Crawl

[anjackson]

As it's always difficult to find shareable files of various formats, one option would be to use the Common Crawl indexes to find relevant items. Common Crawl publish Apache Parquet indexes which can be used for this kind of thing. e.g.

• https://avilpage.com/2022/11/common-crawl-laptop-extract-subset.html (particularly the DuckDB example near the end)
• How to build a subset of CommonCrawl dataset?
• https://github.com/commoncrawl/cc-index-table?tab=readme-ov-file
• https://skeptric.com/reading-parquet-metadata/
• https://www.commoncrawl.org/blog/web-archiving-file-formats-explained
• https://commoncrawl.org/get-started
  • https://data.commoncrawl.org/crawl-data/CC-MAIN-2013-20/index.html (current earliest data with Parquet)
  • https://data.commoncrawl.org/crawl-data/CC-MAIN-2024-10/index.html

Needs thinking through, and understanding if what any costs and impacts are.

Note some prior work that is related:

• https://www.ideals.illinois.edu/items/128320
• https://openpreservation.org/blogs/monitoring-disappearing-file-formats-2/