search

scrapy / protego

A pure-Python robots.txt parser with support for modern conventions.
BSD 3-Clause "New" or "Revised" License
54 stars 28 forks source link

Update for RFC 9309 Robots Exclusion Protocol #35

Open Gallaecio opened 1 year ago

Gallaecio commented 1 year ago

It seems Google got a standard published a few months ago. We should review it in detail, and make sure we follow it. We should also mention it in the README.