Closed bingalls-compete closed 11 years ago
If a site responds with a 301 or 302, and the location has a port number (e.g. http://testsite.com:80/anotherrobots.txt), this will now not crash.
If a site responds with a 301 or 302, and the location has a port number (e.g. http://testsite.com:80/anotherrobots.txt), this will now not crash.