search

spatie / robots-txt

Determine if a page may be crawled from robots.txt, robots meta tags and robot headers
https://spatie.be/en/opensource/php
MIT License
219 stars 36 forks source link

Handle multiple user-agent #29

Closed Kixell-NicolasJardillier closed 3 years ago

Kixell-NicolasJardillier commented 3 years ago

Related to issue : https://github.com/spatie/robots-txt/issues/28

freekmurze commented 3 years ago

Thanks!