Feature/fix parsing robotstxt urls with keywords

spatie / robots-txt

Determine if a page may be crawled from robots.txt, robots meta tags and robot headers

https://spatie.be/en/opensource/php

MIT License

219 stars 36 forks source link

Closed Redominus closed 5 years ago

Redominus commented 5 years ago

Disallowed urls containing disallow word get truncated because str_replace change every ocurrence.

freekmurze commented 5 years ago

Thanks!