Consider *not* respecting robots.txt by default

buren / wayback_archiver

Ruby gem to send URLs to Wayback Machine

https://rubygems.org/gems/wayback_archiver

MIT License

57 stars 11 forks source link

Closed buren closed 2 years ago

buren commented 4 years ago

We could use the same rationale as the Internet Archive has, see this article.

bartman081523 commented 2 years ago

fixed with 1.4 70cc2663d4afd498728989888397a9f42f4afe90 with DEFAULT_RESPECT_ROBOTS_TXT = false