We discovered a bug where the callback in the RobotsParser constructor gets called multiple times, due to large robots.txt files being returned through the request object in multiple chunks. The solution was to alter the response's data listener to collect the chunks, and add the response's end listener to combine the chunks and then call the callback.
We also found a bug which crashes the parser if a rule in robots.txt is malformed (if it has an unquoted %). And we handle it by simply ignoring the rule (rather than guessing what it is supposed to match).
Both of these problems were caused by Wikipedia's robots.txt file, if you are interested in replicating the bugs.
Hi ekalinin,
We discovered a bug where the callback in the RobotsParser constructor gets called multiple times, due to large robots.txt files being returned through the request object in multiple chunks. The solution was to alter the response's data listener to collect the chunks, and add the response's end listener to combine the chunks and then call the callback.
We also found a bug which crashes the parser if a rule in robots.txt is malformed (if it has an unquoted %). And we handle it by simply ignoring the rule (rather than guessing what it is supposed to match).
Both of these problems were caused by Wikipedia's robots.txt file, if you are interested in replicating the bugs.
Thanks, Steve