search

scrapy / protego

A pure-Python robots.txt parser with support for modern conventions.
BSD 3-Clause "New" or "Revised" License
54 stars 28 forks source link

Support for visit-time #39

Closed rasmusson closed 1 year ago

rasmusson commented 1 year ago

I have a site using the visit-time attribute in their robots.txt. Any chance support for this could be implemented? If not are there any suggestions on how to work around this in scrapy?

rasmusson commented 1 year ago

Nice, thanks!