Handle sitemaps that come before any user agent directives

scrapy / protego

A pure-Python robots.txt parser with support for modern conventions.

BSD 3-Clause "New" or "Revised" License

54 stars 28 forks source link

Closed sseveran closed 3 years ago

sseveran commented 3 years ago

Some robots.txt files have Sitemap directives that come before any User-Agent directives. This PR adds support for that.