Add ability to read sitemap files locally

agude / wayback-machine-archiver

A Python script to submit web pages to the Wayback Machine for archiving.

MIT License

71 stars 11 forks source link

Closed kevinmcmahon closed 4 years ago

kevinmcmahon commented 4 years ago

You can load sitemaps via HTTP requests and load URLs via files. This PR lets you read a sitemap from a file.

You can also pass a sitemap.xml file (requires the file:// prefix) to the archiver:

archiver --sitemaps file://sitemap.xml

Added code that first checks if the URL/URI passed the sitemap argument is a file path and attempts to read the file rather than fetch it via the network.
update README with details on archiving sitemap.xml file

agude commented 4 years ago

Thanks for the code @kevinmcmahon! :1st_place_medal:

Merged and released as 1.7.0 with some minor changes.