search

spatie / robots-txt

Determine if a page may be crawled from robots.txt, robots meta tags and robot headers
https://spatie.be/en/opensource/php
MIT License
219 stars 36 forks source link

Robots.txt "Disallow" URI matching should be case-sensitive #45

Closed mattfo0 closed 2 weeks ago

mattfo0 commented 3 weeks ago

According to google robots.txt spec URI matches against disallow rules must be case-sensitive.

The value of the disallow rule is case-sensitive.