Fix Crawl-Delay issue - Githubissues

bopoda / robots-txt-parser

PHP class for parse all directives from robots.txt files according to specifications

http://robots.jeka.by

MIT License

44 stars 17 forks source link

Closed bopoda closed 3 years ago

bopoda commented 3 years ago

Crawl-Delay: 0

Return rule with "0" value instead of ignoring directive