Some robots.txt files for robustness testing

bopoda / robots-txt-parser

PHP class for parse all directives from robots.txt files according to specifications

http://robots.jeka.by

MIT License

44 stars 17 forks source link

Closed LeMoussel closed 7 years ago

LeMoussel commented 7 years ago

For robustness testing,

Confused robots.txt files: http://www.cooljobscanada.com/robots.txt http://www.crowndiamond.org/robots.txt http://ramast.divagaciones.com/robots.txt

Big robots.txt file: http://www.goldmansachs.com/robots.txt

bopoda commented 7 years ago

Confused robots.txt files: http://robots.jeka.by/cooljobscanada.com (parsed) http://robots.jeka.by/crowndiamond.org (parsed with empty rules - seems as bug which depends with encoding) http://robots.jeka.by/ramast.divagaciones.com (parsed with empty rules but it is correctly)

Big robots.txt file: http://robots.jeka.by/goldmansachs.com (parsed during less than 0.5 sec now)