fizx / robots

robots.txt parser
40 stars 17 forks source link

Only test the first element of status #9

Open postmodern opened 8 years ago

postmodern commented 8 years ago
juliend2 commented 4 years ago

And in Webrick, for some reasons, the "OK" is actually "OK " (trailing space). So I think it's better not to rely on that "OK" to be always properly formatted or there at all.