MichaCo / DnsClient.NET

DnsClient.NET is a simple yet very powerful and high performant open source library for the .NET Framework to do DNS lookups
https://dnsclient.michaco.net
Apache License 2.0
781 stars 137 forks source link

Hi rate of garbage collection #168

Closed msoler8785 closed 2 years ago

msoler8785 commented 2 years ago

After we upgraded from 1.3.2 to 1.6.X we noticed our service spent a lot of time in garbage collection. I believe it is related to the UDP client pooling being removed.

Removing UDP client pooling

I think pooling is essential to high performance, but I understand the problems stated in this issue: https://github.com/MichaCo/DnsClient.NET/issues/132

Once we removed the DnsClient from our service you can see the reduction in % time in GC metric in this graph: image

MichaCo commented 2 years ago

How many queries are you doing though? That must be in 10k per second or so? Are you using caching at all?

You might also want to try TCP only now if that works with your DNS server. Performance testing locally works better with TCP connections because I still reuse those.

I'm not really sure we can do much tbh, without investing more time in testing and figuring out how to exactly reuse UDP sockets. I'd rather have it run stable all the time then faster

msoler8785 commented 2 years ago

How many queries are you doing though? That must be in 10k per second or so?

I don't have any firm numbers but it couldn't be more then 200-300 per second, and that would be at peak. They are all TXT queries so maybe that is also contributing to the high GC pressure as the UdpClient probably has to allocate larger buffers?

Are you using caching at all?

Yes there is additional layer of caching on top of the LookupClient but these are the settings I was using:

 var dnsOptions = new LookupClientOptions(dnsServers)
 {
     Timeout = TimeSpan.FromMilliseconds(4000),
     Retries = dnsRetries,
     ContinueOnDnsError = false,
     UseCache = true,
     UseTcpFallback = false,
     CacheFailedResults = true,
     FailedResultsCacheDuration = TimeSpan.FromSeconds(30),
     MaximumCacheTimeout = TimeSpan.FromHours(4),
     MinimumCacheTimeout = TimeSpan.FromHours(1)
 };

You might also want to try TCP only now if that works with your DNS server. Performance testing locally works better with TCP connections because I still reuse those.

I considered trying this. When I get time ill try it out.

I'm not really sure we can do much tbh, without investing more time in testing and figuring out how to exactly reuse UDP sockets. I'd rather have it run stable all the time then faster

I generally agree with this, and I will try to propose some solutions.

msoler8785 commented 2 years ago

I apologize as I may have jumped the gun here. It looks like it was another library from Microsoft that was updated at the same time that was actually causing the issue.

Once I downgraded System.Memory from 4.5.5 to 4.5.4 the issues went away.