Open r0wbrt opened 6 years ago
The back off algorithm has really helped to solve this problem when the server is under light usage by a single client. Before, these errors would occur under light usage from a single client. c0907d42e558c3b4692eabfa334dd197e7061114
See issue https://github.com/golang/go/issues/18588, apparently resolving a moderate number of dns queries causes the resolver to consume all of the file descriptors of the program. To reproduce, open three major news sites via news.google.com at the same time. You will get a list of errors. This is further aggravated by the loop back and localhost connection guards which also do a DNS lookup. So each HTTP request is actually performing at a min 3 dns lookups. Preliminary experimentation where the URL's host name is replaced by the IP and the web server host header is set to the original hostname did not work. The TLS library seems to use the host name in the URL field for validation.
Suggest Fix : Implement a custom resolver that uses a single port (And hence a single OS handle) to do the DNS resolving. Probably could fork the Go resolver for this. Also add some sort of DNS caching into the mix.