search

spatie / robots-txt

Determine if a page may be crawled from robots.txt, robots meta tags and robot headers
https://spatie.be/en/opensource/php
MIT License
219 stars 36 forks source link

Match meta tag for single quotes, often used by Wordpress sites #35

Closed gjportegies closed 2 years ago

freekmurze commented 2 years ago

Thanks!