search

temoto / robotstxt

The robots.txt exclusion protocol implementation for Go language
MIT License
269 stars 55 forks source link

empty disallow is allow #4

Closed igrigorik closed 13 years ago

igrigorik commented 13 years ago

"disallow: " is allow all, as documented on: http://www.robotstxt.org/robotstxt.html

temoto commented 13 years ago

Thank you again :) i should have done that important case myself.