The code for parsing robots.txt introduced in commit 336f693 can run into problems if the content is not in the right form:
If the file is empty, so too will be @lines, so lc will throw an uninitialized value warning at line 84.
If the file does not contain a "User-agent: *" line, the code will get stuck in an infinite loop.
If the file does not contain blank line after "User-agent: *", the pattern match at line 89 will throw an uninitialized value warning when the loop terminates.
The code for parsing robots.txt introduced in commit 336f693 can run into problems if the content is not in the right form:
@lines, solcwill throw an uninitialized value warning at line 84.These problems are resolved in this commit.