Recently we encountered a situation where, due to a misconfigured server, a request to a robots.txt file resulted in an infinite redirect loop between the http and https protocols. This PR adds a guard for these kinds of situations. I followed Google's specification in deciding how to deal with this:
Redirects will generally be followed until a valid result can be found (or a loop is recognized). We will follow a limited number of redirect hops (RFC 1945 for HTTP/1.0 allows up to 5 hops) and then stop and treat it as a 404.
Recently we encountered a situation where, due to a misconfigured server, a request to a robots.txt file resulted in an infinite redirect loop between the
http
andhttps
protocols. This PR adds a guard for these kinds of situations. I followed Google's specification in deciding how to deal with this: