Rule: host not handled - Githubissues

bopoda / robots-txt-parser

PHP class for parse all directives from robots.txt files according to specifications

http://robots.jeka.by

MIT License

44 stars 17 forks source link

Closed LeMoussel closed 7 years ago

LeMoussel commented 7 years ago

robots host directive is not recognized.

User-agent: * Disallow: /search/ Disallow: /goto? Disallow: /login Disallow: /api Host: https://site.com Sitemap: https://site.com/sitemap.xml

Reference : Yandex Host directive Wikipedia Robots exclusion standard - Host

I would make a PR on that.

LeMoussel commented 7 years ago

As much for me. It's OK, eg tests/Host.php