spatie / robots-txt

Determine if a page may be crawled from robots.txt, robots meta tags and robot headers
https://spatie.be/en/opensource/php
MIT License
219 stars 36 forks source link

Code review #2

Closed freekmurze closed 6 years ago

freekmurze commented 6 years ago