search

temoto / robotstxt

The robots.txt exclusion protocol implementation for Go language
MIT License
269 stars 55 forks source link

'Host' directive parsing added. It's often used in Yandex to specify … #14

Closed brainm closed 9 years ago

brainm commented 9 years ago

…mirror, read more https://help.yandex.com/webmaster/controlling-robot/robots-txt.xml#host

temoto commented 9 years ago

Thank you very much, I also use this directive in a robots.txt file.