search

samclarke / robots-parser

NodeJS robots.txt parser with support for wildcard (*) matching.
MIT License
147 stars 18 forks source link

Empty disallow statements are interpreted as disallow all #2

Closed guiweber closed 9 years ago

guiweber commented 9 years ago

Empty disallow statements Disallow: are interpreted as disallow all while they should be interpreted as allow all.

https://en.wikipedia.org/wiki/Robots_exclusion_standard#Examples

guiweber commented 9 years ago

Turns out the empty disallow statement was not the issue.