search

fizx / robots

robots.txt parser
40 stars 17 forks source link

Parsing Error #1

Closed rb2k closed 14 years ago

rb2k commented 14 years ago

This doesn't seem to work:

User-agent: * Disallow:

(found here: http://www.gelsenclan.de/robots.txt )
Having a robots.txt like this will result in NO part of the page being allowed. The expected behavior would be that everything is allowed

fizx commented 14 years ago

Fixed in 0.9.0

rb2k commented 14 years ago

thanks!