It would be nice if the "robots.txt request timed out" would tell for which uri it actually failed.
This is the method in question:
def self.get_robots_txt(uri, user_agent)
begin
Timeout::timeout(Robots.timeout) do
io = URI.join(uri.to_s, "/robots.txt").open("User-Agent" => user_agent) rescue nil
end
rescue Timeout::Error
STDERR.puts "robots.txt request timed out"
end
end
It would be nice if the "robots.txt request timed out" would tell for which uri it actually failed. This is the method in question: