Optionally use hash for deduplication with the Wayback Machine

ArchiveTeam / urls-grab

Archiving URLs (outlinks) from a variety of sources.

The Unlicense

15 stars 5 forks source link

Optionally use hash for deduplication with the Wayback Machine #6

Open Arkiver2 opened 3 years ago

Arkiver2 commented 3 years ago

This is an extension on https://github.com/ArchiveTeam/urls-grab/issues/5 where we now also check the hashes from the Wayback Machine record to check if the payload of the record is actually equal.