search

spatie / robots-txt

Determine if a page may be crawled from robots.txt, robots meta tags and robot headers
https://spatie.be/en/opensource/php
MIT License
219 stars 36 forks source link

Fix nofollow or noindex check for Headers #11

Closed RobinDev closed 5 years ago

RobinDev commented 5 years ago

If a headers send X-Robots-Tag: nofollow and then i check for mayFollow('PrivateBot') I get true. I think it's not the expected result.

Pull-requesting in five minuts.

spatie-bot commented 5 years ago

Dear contributor,

because this issue seems to be inactive for quite some time now, I've automatically closed it. If you feel this issue deserves some attention from my human colleagues feel free to reopen it.