search

agude / wayback-machine-archiver

A Python script to submit web pages to the Wayback Machine for archiving.
https://pypi.org/project/wayback-machine-archiver/
MIT License
71 stars 11 forks source link

Deduplicate the URL list before submitting it #13

Closed agude closed 4 years ago

agude commented 4 years ago

The Internet Archive only allows one backup per 10 minutes, so we shouldn't submit duplicate URLs.