search

socketry / async-dns

An asynchronous DNS resolver and server.
MIT License
96 stars 14 forks source link

Slow performance with passthrough! under high load #12

Closed phallguy closed 2 years ago

phallguy commented 4 years ago

We are running a DNS filter based on async-dns in production serving ~8K requests per second at peak. Under medium load we see fantastic response times under 10ms. At around 3.5-5K rps we see about a 1.5-2 second delay when we call passthrough! to the upstream resolver.

During that same time attempting to contact the upstream server directly with nslookup/dig we see response times of less than 10ms.

CPU does increase - but settles around 15% utilization which seems reasonable and not the source of the bottleneck. Memory doesn't change significantly and remains steady across load.

Do you have any ideas about what might cause the delay when calling the upstream resolver from our app? Are there any connection limits or concurrency limits we could tweak?

ioquatix commented 4 years ago

@phallguy I am happy to offer commercial support if this is for a business related issue. Let me know if this is something you'd like to explore.

Regarding latency under load, there are some situations where the upstream server might be causing issues, or maybe it's an issue with the Ruby code.

If you enable debug logging we could capture the output from https://github.com/socketry/async-dns/blob/bee5fd678b9c20b0f1ac20b2e9a6cabccbad92b0/lib/async/dns/resolver.rb#L136-L168

Regarding settings to tweak, there are very few settings you can tweak in async and that in and of itself is a design goal. Performance should be the best it can be out of the box with no changes.

Even if there are intrinsic performance limitations, there is nothing stopping you using async-container and scaling this up to as many CPU cores as you have available.

ioquatix commented 2 years ago

@phallguy If you could update and/or close this issue that would be much appreciated.