Run this command, go run . -url google.com -robots -plain > results.txt,
can not found /m/finance in results, but it's in https://www.google.com/robots.txt.
This is because some lines in robots.txt have more than one white space,
function recordIfInScope parse the url and throw an error, so it' not in the result.
Trim it in function parseRobots will fix this bug.
hakluke
commented
3 years ago
Run this command,
go run . -url google.com -robots -plain > results.txt, can not found/m/financein results, but it's inhttps://www.google.com/robots.txt.This is because some lines in robots.txt have more than one white space, function
recordIfInScopeparse the url and throw an error, so it' not in the result.Trim it in function
parseRobotswill fix this bug.