search

spatie / robots-txt

Determine if a page may be crawled from robots.txt, robots meta tags and robot headers
https://spatie.be/en/opensource/php
MIT License
219 stars 36 forks source link

Fix Wildcard check in Robots.txt and Headers #12

Closed RobinDev closed 5 years ago

RobinDev commented 5 years ago

Checking with a Wildcard failed in RobotsHeaders (i didn't yet check for robots.txt but I suppose it's failing to).

https://developers.google.com/search/reference/robots_txt#order-of-precedence-for-user-agents

Pull-request in five minutes.

spatie-bot commented 5 years ago

Dear contributor,

because this issue seems to be inactive for quite some time now, I've automatically closed it. If you feel this issue deserves some attention from my human colleagues feel free to reopen it.