search

bopoda / robots-txt-parser

PHP class for parse all directives from robots.txt files according to specifications
http://robots.jeka.by
MIT License
44 stars 17 forks source link

Crawl Delay of 0 will get ignored #59

Closed offsky closed 3 years ago

offsky commented 3 years ago

If the robots.txt file has a directive of Crawl-Delay: 0 the RobotTxtParser will return an array of rules that excludes this directive. I realize that having a crawl delay of 0 is the same thing as omitting the directive, so this may be a minor squabble, but it would be nice to return all the valid rules even if the crawler would ignore them.

Simple fix: In handleDirective change if (!$value) { to if (!strlen($value)) { or if ($value==='') {

bopoda commented 3 years ago

@offsky Thanks for reporting the issue. I agree that's better to return a rule with a "0" value, the more obvious the better. Should be fixed by PR https://github.com/bopoda/robots-txt-parser/pull/62