Batch-Downloading a collection of snapshots via CDX

iipc / openwayback

The OpenWayback Development

http://www.netpreserve.org/openwayback

Apache License 2.0

486 stars 275 forks source link

Batch-Downloading a collection of snapshots via CDX #331

Open fbuchinger opened 7 years ago

fbuchinger commented 7 years ago

Hi,

regarding the use cases #14 (Buld & Batch Requests) and #16 (WARC file management) mentioned in https://github.com/iipc/openwayback/wiki/CDX-Server-requirements: is anything of that implemented yet? If yes, is it already in production at the Wayback Machine?

For a research project, we need to analyze snapshots of a small set of webpages in a given timeframe. It would be great if the snapshots could be downloaded to speed up analysis.

Best,

Franz

ldko commented 7 years ago

Neither bulk/batch requests nor the (W)ARC file management described have been implemented here.