TechnitiumSoftware / DnsServer

Technitium DNS Server
https://technitium.com/dns/
GNU General Public License v3.0
4.25k stars 418 forks source link

The latest version of cpu takes up very high #168

Closed liang-hiwin closed 3 years ago

liang-hiwin commented 4 years ago

The latest version of DnsServer occupies a lot of CPU.

PY NL7C0(8 7 LTPW_N(~UG

ShreyasZare commented 4 years ago

Thanks for the feedback. Please share the DNS server logs and the stats on the dashboard.

ShreyasZare commented 4 years ago

From the logs it looks like there is some app on your machine which is querying "interface.biliapi.com" for some unknown type (65) continuously. This probably is causing the DNS server to use up CPU as it tries to resolve the domain which is failing.

liang-hiwin commented 4 years ago

From the logs it looks like there is some app on your machine which is querying "interface.biliapi.com" for some unknown type (65) continuously. This probably is causing the DNS server to use up CPU as it tries to resolve the domain which is failing.

Is it EDNS request. Or DNSSEC request

ShreyasZare commented 4 years ago

Is it EDNS request. Or DNSSEC request

Its neither of them.

liang-hiwin commented 4 years ago

Is it EDNS request. Or DNSSEC request

Its neither of them.

What caused it

ShreyasZare commented 4 years ago

What caused it

Some app that is installed is doing those queries.

muhanadali100 commented 4 years ago

The latest version of DnsServer occupies a lot of CPU.

PY NL7C0(8 7 LTPW_N(~UG

could you tell me how you showing that's please ??

liang-hiwin commented 4 years ago

The latest version of DnsServer occupies a lot of CPU. PY NL7C0(8 7 LTPW_N(~UG

could you tell me how you showing that's please ??

htop

muhanadali100 commented 4 years ago

The latest version of DnsServer occupies a lot of CPU. PY NL7C0(8 7 LTPW_N(~UG

could you tell me how you showing that's please ??

htop

Desktop Screenshot 2020 09 13 - 11 11 44 41

thanx

ShreyasZare commented 4 years ago

From the logs it looks like there is some app on your machine which is querying "interface.biliapi.com" for some unknown type (65) continuously. This probably is causing the DNS server to use up CPU as it tries to resolve the domain which is failing.

There is a post on unknown type 65 on dns-operations mailing list: https://lists.dns-oarc.net/pipermail/dns-operations/2020-September/020506.html

liang-hiwin commented 4 years ago

从日志看来,您的计算机上似乎有一些应用程序正在不断查询“ interface.biliapi.com”中某些未知类型(65)。这可能导致DNS服务器在尝试解析失败的域时耗尽CPU。

dns-operations邮件列表上有一篇关于未知类型65的帖子:https : //lists.dns-oarc.net/pipermail/dns-operations/2020-September/020506.html

thx

askar882 commented 4 years ago

I'm facing the same problem as well. The application making frequent DNS query might be qBittorrent in my case. As there are some outdated torrents with trackers which doesn't exist any longer, qBittorrent is repeatedly making DNS queries causing DNS Server high CPU usage.

ShreyasZare commented 4 years ago

I'm facing the same problem as well. The application making frequent DNS query might be qBittorrent in my case. As there are some outdated torrents with trackers which doesn't exist any longer, qBittorrent is repeatedly making DNS queries causing DNS Server high CPU usage.

Thanks for the feedback. Is the CPU being used continuously or are there some peaks occasionally?

Please let me know the hardware config running the DNS server. Also let me know the stats you see on the dashboard so understand the load.

The latest version can handle high loads quite easily and so it doesn't seem that some app querying frequently will cause this.

askar882 commented 4 years ago

I'm facing the same problem as well. The application making frequent DNS query might be qBittorrent in my case. As there are some outdated torrents with trackers which doesn't exist any longer, qBittorrent is repeatedly making DNS queries causing DNS Server high CPU usage.

Thanks for the feedback. Is the CPU being used continuously or are there some peaks occasionally?

Please let me know the hardware config running the DNS server. Also let me know the stats you see on the dashboard so understand the load.

The latest version can handle high loads quite easily and so it doesn't seem that some app querying frequently will cause this.

I'm running DNS server on my laptop with Intel(R) Core(TM) i5-8265U CPU and an 8G DDR4 RAM. The CPU is being used continuously once this issue occurs. DNS service takes up about 20% of the CPU and it remains until I manually restart the service.

liang-hiwin commented 4 years ago

I'm facing the same problem as well. The application making frequent DNS query might be qBittorrent in my case. As there are some outdated torrents with trackers which doesn't exist any longer, qBittorrent is repeatedly making DNS queries causing DNS Server high CPU usage.

Thanks for the feedback. Is the CPU being used continuously or are there some peaks occasionally? Please let me know the hardware config running the DNS server. Also let me know the stats you see on the dashboard so understand the load. The latest version can handle high loads quite easily and so it doesn't seem that some app querying frequently will cause this.

I'm running DNS server on my laptop with Intel(R) Core(TM) i5-8265U CPU and an 8G DDR4 RAM. The CPU is being used continuously once this issue occurs. DNS service takes up about 20% of the CPU and it remains until I manually restart the service.

me too

ShreyasZare commented 4 years ago

Thanks @askar882 and @thb007 for the feedback. I am running the server on 3 different hardware including raspberry pi and haven't seen any CPU usage issue.

I need to try to reproduce this on my setup only then I can find out whats causing the issue. Please let me know the following details:

If possible, do share a zip copy of the config folder on your server so that I can use the same config on the test setup to try to reproduce this issue. You may exclude the logs folder inside the config if you don't wish to share it though it will be very useful for analysis. Do share the zip on email.

liang-hiwin commented 4 years ago

Thanks @askar882 and @thb007 for the feedback. I am running the server on 3 different hardware including raspberry pi and haven't seen any CPU usage issue.

I need to try to reproduce this on my setup only then I can find out whats causing the issue. Please let me know the following details:

  • Are you running the DNS server on a public IP address accessible on the Internet?
  • Are you hosting any live domain names on the DNS server?
  • If you are using it as a resolver, are you using any forwarders (DoH or DoT) or using without a forwarder?
  • What OS are you running the DNS server on?

If possible, do share a zip copy of the config folder on your server so that I can use the same config on the test setup to try to reproduce this issue. You may exclude the logs folder inside the config if you don't wish to share it though it will be very useful for analysis. Do share the zip on email.

I use it as a public udp dns forwarder. Of course, recursive analysis is also checked in the settings. cpu sometimes still occupies 100%

askar882 commented 4 years ago

Thanks @askar882 and @thb007 for the feedback. I am running the server on 3 different hardware including raspberry pi and haven't seen any CPU usage issue.

I need to try to reproduce this on my setup only then I can find out what's causing the issue. Please let me know the following details:

  • Are you running the DNS server on a public IP address accessible on the Internet?
  • Are you hosting any live domain names on the DNS server?
  • If you are using it as a resolver, are you using any forwarders (DoH or DoT) or using it without a forwarder?
  • What OS are you running the DNS server on?

If possible, do share a zip copy of the config folder on your server so that I can use the same config on the test setup to try to reproduce this issue. You may exclude the logs folder inside the config if you don't wish to share it though it will be very useful for analysis. Do share the zip on email.

I'm using the DNS server as a local resolver and I just added Google DNS and OpenDNS to the forwarders list with DNS-over-UDP selected. I'm running the DNS server on Windows 10 Pro version 2004. Here is part of the log file generated during the issue. I hope this helps.

ShreyasZare commented 4 years ago

I'm using the DNS server as a local resolver and I just added Google DNS and OpenDNS to the forwarders list with DNS-over-UDP selected. I'm running the DNS server on Windows 10 Pro version 2004. Here is part of the log file generated during the issue. I hope this helps.

Thanks for the logs. It seems that some app is querying invalid domain name causing errors. Continuous generation of error will increase CPU usage which is normal.

ShreyasZare commented 4 years ago

I use it as a public udp dns forwarder. Of course, recursive analysis is also checked in the settings. cpu sometimes still occupies 100%

Please share logs for the duration when CPU is getting used up.

liang-hiwin commented 4 years ago

I use it as a public udp dns forwarder. Of course, recursive analysis is also checked in the settings. cpu sometimes still occupies 100%

Please share logs for the duration when CPU is getting used up. Do I need to open a detailed log?

ShreyasZare commented 4 years ago

Do I need to open a detailed log?

Just the DNS server log file that you see in the web console Log tab.

askar882 commented 3 years ago

Technitium CPU usage Here it comes again. The DNS Server is taking up to 25% of my CPU. I looked through the log and I didn't find any exceptions like last time. All I found is some failed DNS queries which the server returned Name Error. Here is a part of the log. The real problem bothering me is that the DNS Server continuously takes up lots of CPU once the error is somehow triggered. And the CPU usage won't get back to normal until I restart or stop the DNS service using Service Manager. It is totally okay with me if the high CPU usage issue would just continue for less than10 minutes. But from what I see, it won't recover automatically forever.

askar882 commented 3 years ago

threads stack This is what I found using Process Explorer. The second picture is the stack of the first thread.

ShreyasZare commented 3 years ago

@askar882 thanks for the details. I know I am asking too much data. Can you try to capture the network traffic with Wireshark or tcpdump during the high CPU event and send the pcap file? This will help to understand if there is any kind of resolution loop that is going on at the network level.

ShreyasZare commented 3 years ago

@askar882 thanks for the stack trace from process explorer which gave the clue of where to look for the bug.

I have found one issue that can cause the thread to keep looping. It occurs when someone on the network tries to spoof a DNS response with a UDP packet but with a different source IP address than was expected. This causes the DNS server to reject that response but then the read task state is not reset and the while loop keeps reading the same packet over and over again. This could potentially mean that some entity on your network is trying to do DNS cache poisoning attack.

If you could manage to capture network traffic in a pcap file then this case can be confirmed and it would be clear if its the same bug or something different.

liang-hiwin commented 3 years ago

@askar882 thanks for the stack trace from process explorer which gave the clue of where to look for the bug.

I have found one issue that can cause the thread to keep looping. It occurs when someone on the network tries to spoof a DNS response with a UDP packet but with a different source IP address than was expected. This causes the DNS server to reject that response but then the read task state is not reset and the while loop keeps reading the same packet over and over again. This could potentially mean that some entity on your network is trying to do DNS cache poisoning attack.

If you could manage to capture network traffic in a pcap file then this case can be confirmed and it would be clear if its the same bug or something different.

If a non-existent domain name is maliciously flushed DNS cache, will it cause excessive CPU usage?

ShreyasZare commented 3 years ago

If a non-existent domain name is maliciously flushed DNS cache, will it cause excessive CPU usage?

It does not depend on the domain name in query.

Lets take an example: the DNS server receives a query for example.com, there is no cache entry for it so, it forwards the query to lets say 8.8.8.8. Now, before the response from 8.8.8.8 arrives, there is another response coming from 192.168.100.1 (just a made up IP address for this example, could be any IP address). The response's IP address does not match with the IP address the request was forwarded to so, the DNS server will skip this response and should read the next response. The bug here is that the receive task that reads response is not getting reset so the old response is still there which keeps on getting read by the server causing a loop that does not exit causing CPU usage.

I have tested this kind of attack and as expected there is continuous CPU usage.

askar882 commented 3 years ago

@askar882 thanks for the stack trace from process explorer which gave the clue of where to look for the bug.

I have found one issue that can cause the thread to keep looping. It occurs when someone on the network tries to spoof a DNS response with a UDP packet but with a different source IP address than was expected. This causes the DNS server to reject that response but then the read task state is not reset and the while loop keeps reading the same packet over and over again. This could potentially mean that some entity on your network is trying to do a DNS cache poisoning attack.

If you could manage to capture network traffic in a pcap file then this case can be confirmed and it would be clear if its the same bug or something different.

Sorry for the late reply. It seems that you are suspecting that someone is performing a DNS Spoofing attack, which I don't think possible. Because my laptop running DNS service is behind a NAT and I also blocked access to the DNS Server in my firewall. So the DNS Server is only accessible from my localhost. I've created Minidump using Process Explorer, and I will send it to you if you think it helps.

ShreyasZare commented 3 years ago

Sorry for the late reply. It seems that you are suspecting that someone is performing a DNS Spoofing attack, which I don't think possible. Because my laptop running DNS service is behind a NAT and I also blocked access to the DNS Server in my firewall. So the DNS Server is only accessible from my localhost. I've created Minidump using Process Explorer, and I will send it to you if you think it helps.

Thanks for the details. It would be best if you can capture network traffic as a pcap file when this event occurs using Wireshark. This will give much better picture to understand the issue.

askar882 commented 3 years ago

Thanks for the details. It would be best if you can capture network traffic as a pcap file when this event occurs using Wireshark. This will give a much better picture to understand the issue.

Ok, I'll try to capture network traffic once the issue recurs.

ShreyasZare commented 3 years ago

@askar882 have you captured the pcap file? let me know if you have any issues.

Kevin-Andrew commented 3 years ago

@ShreyasZare , I also experience times when the DNS Server appears to get into some kind of infinite loop and uses essentially 100% CPU until I kill the process and restart it.

I have attached the debugger when this happens, but can't figure out what is causing the problem.

I have the setting "Log All Queries" enabled, and it doesn't appear that the log grows during this time. Meaning, it doesn't appear to be some other external program making millions of DNS queries. But I could be wrong.

I see your request to askar882 about capturing network traffic when the problem happens. Unfortunately, I wouldn't be comfortable installing Wireshark on my machine.

However, I'd be happy to try and contact you directly next time it happens and share my computer screen with you such that you could attempt to debug it.

It looks like you are in Mumbai, India, and I am in the United States. So there may be difficulties with time differences.

Kevin-Andrew commented 3 years ago

Just had the problem happen again. I'd also be happy to send you the log file and my config.

Let me know.

ShreyasZare commented 3 years ago

@Kevin-Andrew thanks for the feedback. The issue is not visible in log so its difficult to debug it.

I have next update coming up on weekend which fixes one bug that I found which I was able to reproduce with same symptoms. I think this is the bug that is causing the issue as I have not found any thing else till now. Lets see if the problem stops occurring for everyone with this new update.

Kevin-Andrew commented 3 years ago

Thanks, @ShreyasZare . Will do.

ShreyasZare commented 3 years ago

New version 5.4 is now available which should fix this issue. Please reopen this if issue persists.

Rustem05 commented 3 years ago

Dns ???