optionally allow partial matches and global groups

spatie / robots-txt

Determine if a page may be crawled from robots.txt, robots meta tags and robot headers

https://spatie.be/en/opensource/php

MIT License

219 stars 36 forks source link

optionally allow partial matches and global groups #43

Closed resohead closed 5 months ago

resohead commented 5 months ago

This package currently matches a given user-agent with robots.txt exactly. This PR allows developers to use partial matches or exact matches (exactMatchesOnly) and also to consider '*' user-agent when checking if a path is denied (ignoreGlobalGroup).

The original behaviour remains unchanged by default, existing tests pass and additional tests added for partial matches and global groups.

freekmurze commented 5 months ago

Thanks!