search

webignition / robots-txt-file

Models a robots.txt file
MIT License
18 stars 3 forks source link

Feature request: isAllowed() #5

Closed greendrake closed 7 years ago

greendrake commented 8 years ago

I am building a considerate bot which does not really care about directives and sitemaps. All it needs to know is whether or not it is allowed to access a certain path. Is it possible to use this software in that fashion already? For example, something like:

$parser = new \webignition\RobotsTxt\File\Parser();
$parser->setContent(file_get_contents('http://example.com/robots.txt'));
$parser->setUserAgent('MyBot');
var_dump($parser->isAllowed('/foo'));
hbea2014 commented 8 years ago

+1 Same here :)

LeMoussel commented 8 years ago

:+1:

webignition commented 7 years ago

@greendrake That's a great idea, I'd certainly like to implement that soon.

webignition commented 7 years ago

This is now present in the 2.0 release

See example at https://github.com/webignition/robots-txt-file/blob/master/README.md#check-if-a-user-agent-is-allowed-to-access-a-url-path