From ArchiveBot issue https://github.com/ArchiveTeam/ArchiveBot/issues/169
This would be helpful to have in grab-site, too, e.g. for sites with many large files a few of which change regularly — it would be nice to be able to supply a CDX of a previous crawl, and skip things that were already gotten....
From ArchiveBot issue https://github.com/ArchiveTeam/ArchiveBot/issues/169 This would be helpful to have in grab-site, too, e.g. for sites with many large files a few of which change regularly — it would be nice to be able to supply a CDX of a previous crawl, and skip things that were already gotten....