Closed liang-hiwin closed 3 years ago
Thanks for the feedback. Please share the DNS server logs and the stats on the dashboard.
From the logs it looks like there is some app on your machine which is querying "interface.biliapi.com" for some unknown type (65) continuously. This probably is causing the DNS server to use up CPU as it tries to resolve the domain which is failing.
From the logs it looks like there is some app on your machine which is querying "interface.biliapi.com" for some unknown type (65) continuously. This probably is causing the DNS server to use up CPU as it tries to resolve the domain which is failing.
Is it EDNS request. Or DNSSEC request
Is it EDNS request. Or DNSSEC request
Its neither of them.
Is it EDNS request. Or DNSSEC request
Its neither of them.
What caused it
What caused it
Some app that is installed is doing those queries.
The latest version of DnsServer occupies a lot of CPU.
could you tell me how you showing that's please ??
The latest version of DnsServer occupies a lot of CPU.
could you tell me how you showing that's please ??
htop
The latest version of DnsServer occupies a lot of CPU.
could you tell me how you showing that's please ??
htop
thanx
From the logs it looks like there is some app on your machine which is querying "interface.biliapi.com" for some unknown type (65) continuously. This probably is causing the DNS server to use up CPU as it tries to resolve the domain which is failing.
There is a post on unknown type 65 on dns-operations mailing list: https://lists.dns-oarc.net/pipermail/dns-operations/2020-September/020506.html
从日志看来,您的计算机上似乎有一些应用程序正在不断查询“ interface.biliapi.com”中某些未知类型(65)。这可能导致DNS服务器在尝试解析失败的域时耗尽CPU。
dns-operations邮件列表上有一篇关于未知类型65的帖子:https : //lists.dns-oarc.net/pipermail/dns-operations/2020-September/020506.html
thx
I'm facing the same problem as well. The application making frequent DNS query might be qBittorrent in my case. As there are some outdated torrents with trackers which doesn't exist any longer, qBittorrent is repeatedly making DNS queries causing DNS Server high CPU usage.
I'm facing the same problem as well. The application making frequent DNS query might be qBittorrent in my case. As there are some outdated torrents with trackers which doesn't exist any longer, qBittorrent is repeatedly making DNS queries causing DNS Server high CPU usage.
Thanks for the feedback. Is the CPU being used continuously or are there some peaks occasionally?
Please let me know the hardware config running the DNS server. Also let me know the stats you see on the dashboard so understand the load.
The latest version can handle high loads quite easily and so it doesn't seem that some app querying frequently will cause this.
I'm facing the same problem as well. The application making frequent DNS query might be qBittorrent in my case. As there are some outdated torrents with trackers which doesn't exist any longer, qBittorrent is repeatedly making DNS queries causing DNS Server high CPU usage.
Thanks for the feedback. Is the CPU being used continuously or are there some peaks occasionally?
Please let me know the hardware config running the DNS server. Also let me know the stats you see on the dashboard so understand the load.
The latest version can handle high loads quite easily and so it doesn't seem that some app querying frequently will cause this.
I'm running DNS server on my laptop with Intel(R) Core(TM) i5-8265U CPU and an 8G DDR4 RAM. The CPU is being used continuously once this issue occurs. DNS service takes up about 20% of the CPU and it remains until I manually restart the service.
I'm facing the same problem as well. The application making frequent DNS query might be qBittorrent in my case. As there are some outdated torrents with trackers which doesn't exist any longer, qBittorrent is repeatedly making DNS queries causing DNS Server high CPU usage.
Thanks for the feedback. Is the CPU being used continuously or are there some peaks occasionally? Please let me know the hardware config running the DNS server. Also let me know the stats you see on the dashboard so understand the load. The latest version can handle high loads quite easily and so it doesn't seem that some app querying frequently will cause this.
I'm running DNS server on my laptop with Intel(R) Core(TM) i5-8265U CPU and an 8G DDR4 RAM. The CPU is being used continuously once this issue occurs. DNS service takes up about 20% of the CPU and it remains until I manually restart the service.
me too
Thanks @askar882 and @thb007 for the feedback. I am running the server on 3 different hardware including raspberry pi and haven't seen any CPU usage issue.
I need to try to reproduce this on my setup only then I can find out whats causing the issue. Please let me know the following details:
If possible, do share a zip copy of the config
folder on your server so that I can use the same config on the test setup to try to reproduce this issue. You may exclude the logs
folder inside the config
if you don't wish to share it though it will be very useful for analysis. Do share the zip on email.
Thanks @askar882 and @thb007 for the feedback. I am running the server on 3 different hardware including raspberry pi and haven't seen any CPU usage issue.
I need to try to reproduce this on my setup only then I can find out whats causing the issue. Please let me know the following details:
- Are you running the DNS server on a public IP address accessible on the Internet?
- Are you hosting any live domain names on the DNS server?
- If you are using it as a resolver, are you using any forwarders (DoH or DoT) or using without a forwarder?
- What OS are you running the DNS server on?
If possible, do share a zip copy of the
config
folder on your server so that I can use the same config on the test setup to try to reproduce this issue. You may exclude thelogs
folder inside theconfig
if you don't wish to share it though it will be very useful for analysis. Do share the zip on email.
I use it as a public udp dns forwarder. Of course, recursive analysis is also checked in the settings. cpu sometimes still occupies 100%
Thanks @askar882 and @thb007 for the feedback. I am running the server on 3 different hardware including raspberry pi and haven't seen any CPU usage issue.
I need to try to reproduce this on my setup only then I can find out what's causing the issue. Please let me know the following details:
- Are you running the DNS server on a public IP address accessible on the Internet?
- Are you hosting any live domain names on the DNS server?
- If you are using it as a resolver, are you using any forwarders (DoH or DoT) or using it without a forwarder?
- What OS are you running the DNS server on?
If possible, do share a zip copy of the
config
folder on your server so that I can use the same config on the test setup to try to reproduce this issue. You may exclude thelogs
folder inside theconfig
if you don't wish to share it though it will be very useful for analysis. Do share the zip on email.
I'm using the DNS server as a local resolver and I just added Google DNS and OpenDNS to the forwarders list with DNS-over-UDP selected. I'm running the DNS server on Windows 10 Pro version 2004. Here is part of the log file generated during the issue. I hope this helps.
I'm using the DNS server as a local resolver and I just added Google DNS and OpenDNS to the forwarders list with DNS-over-UDP selected. I'm running the DNS server on Windows 10 Pro version 2004. Here is part of the log file generated during the issue. I hope this helps.
Thanks for the logs. It seems that some app is querying invalid domain name causing errors. Continuous generation of error will increase CPU usage which is normal.
I use it as a public udp dns forwarder. Of course, recursive analysis is also checked in the settings. cpu sometimes still occupies 100%
Please share logs for the duration when CPU is getting used up.
I use it as a public udp dns forwarder. Of course, recursive analysis is also checked in the settings. cpu sometimes still occupies 100%
Please share logs for the duration when CPU is getting used up. Do I need to open a detailed log?
Do I need to open a detailed log?
Just the DNS server log file that you see in the web console Log tab.
Here it comes again. The DNS Server is taking up to 25% of my CPU. I looked through the log and I didn't find any exceptions like last time. All I found is some failed DNS queries which the server returned Name Error. Here is a part of the log. The real problem bothering me is that the DNS Server continuously takes up lots of CPU once the error is somehow triggered. And the CPU usage won't get back to normal until I restart or stop the DNS service using Service Manager. It is totally okay with me if the high CPU usage issue would just continue for less than10 minutes. But from what I see, it won't recover automatically forever.
This is what I found using Process Explorer. The second picture is the stack of the first thread.
@askar882 thanks for the details. I know I am asking too much data. Can you try to capture the network traffic with Wireshark or tcpdump during the high CPU event and send the pcap file? This will help to understand if there is any kind of resolution loop that is going on at the network level.
@askar882 thanks for the stack trace from process explorer which gave the clue of where to look for the bug.
I have found one issue that can cause the thread to keep looping. It occurs when someone on the network tries to spoof a DNS response with a UDP packet but with a different source IP address than was expected. This causes the DNS server to reject that response but then the read task state is not reset and the while loop keeps reading the same packet over and over again. This could potentially mean that some entity on your network is trying to do DNS cache poisoning attack.
If you could manage to capture network traffic in a pcap file then this case can be confirmed and it would be clear if its the same bug or something different.
@askar882 thanks for the stack trace from process explorer which gave the clue of where to look for the bug.
I have found one issue that can cause the thread to keep looping. It occurs when someone on the network tries to spoof a DNS response with a UDP packet but with a different source IP address than was expected. This causes the DNS server to reject that response but then the read task state is not reset and the while loop keeps reading the same packet over and over again. This could potentially mean that some entity on your network is trying to do DNS cache poisoning attack.
If you could manage to capture network traffic in a pcap file then this case can be confirmed and it would be clear if its the same bug or something different.
If a non-existent domain name is maliciously flushed DNS cache, will it cause excessive CPU usage?
If a non-existent domain name is maliciously flushed DNS cache, will it cause excessive CPU usage?
It does not depend on the domain name in query.
Lets take an example: the DNS server receives a query for example.com, there is no cache entry for it so, it forwards the query to lets say 8.8.8.8. Now, before the response from 8.8.8.8 arrives, there is another response coming from 192.168.100.1 (just a made up IP address for this example, could be any IP address). The response's IP address does not match with the IP address the request was forwarded to so, the DNS server will skip this response and should read the next response. The bug here is that the receive task that reads response is not getting reset so the old response is still there which keeps on getting read by the server causing a loop that does not exit causing CPU usage.
I have tested this kind of attack and as expected there is continuous CPU usage.
@askar882 thanks for the stack trace from process explorer which gave the clue of where to look for the bug.
I have found one issue that can cause the thread to keep looping. It occurs when someone on the network tries to spoof a DNS response with a UDP packet but with a different source IP address than was expected. This causes the DNS server to reject that response but then the read task state is not reset and the while loop keeps reading the same packet over and over again. This could potentially mean that some entity on your network is trying to do a DNS cache poisoning attack.
If you could manage to capture network traffic in a pcap file then this case can be confirmed and it would be clear if its the same bug or something different.
Sorry for the late reply. It seems that you are suspecting that someone is performing a DNS Spoofing attack, which I don't think possible. Because my laptop running DNS service is behind a NAT and I also blocked access to the DNS Server in my firewall. So the DNS Server is only accessible from my localhost. I've created Minidump using Process Explorer, and I will send it to you if you think it helps.
Sorry for the late reply. It seems that you are suspecting that someone is performing a DNS Spoofing attack, which I don't think possible. Because my laptop running DNS service is behind a NAT and I also blocked access to the DNS Server in my firewall. So the DNS Server is only accessible from my localhost. I've created Minidump using Process Explorer, and I will send it to you if you think it helps.
Thanks for the details. It would be best if you can capture network traffic as a pcap file when this event occurs using Wireshark. This will give much better picture to understand the issue.
Thanks for the details. It would be best if you can capture network traffic as a pcap file when this event occurs using Wireshark. This will give a much better picture to understand the issue.
Ok, I'll try to capture network traffic once the issue recurs.
@askar882 have you captured the pcap file? let me know if you have any issues.
@ShreyasZare , I also experience times when the DNS Server appears to get into some kind of infinite loop and uses essentially 100% CPU until I kill the process and restart it.
I have attached the debugger when this happens, but can't figure out what is causing the problem.
I have the setting "Log All Queries" enabled, and it doesn't appear that the log grows during this time. Meaning, it doesn't appear to be some other external program making millions of DNS queries. But I could be wrong.
I see your request to askar882 about capturing network traffic when the problem happens. Unfortunately, I wouldn't be comfortable installing Wireshark on my machine.
However, I'd be happy to try and contact you directly next time it happens and share my computer screen with you such that you could attempt to debug it.
It looks like you are in Mumbai, India, and I am in the United States. So there may be difficulties with time differences.
Just had the problem happen again. I'd also be happy to send you the log file and my config.
Let me know.
@Kevin-Andrew thanks for the feedback. The issue is not visible in log so its difficult to debug it.
I have next update coming up on weekend which fixes one bug that I found which I was able to reproduce with same symptoms. I think this is the bug that is causing the issue as I have not found any thing else till now. Lets see if the problem stops occurring for everyone with this new update.
Thanks, @ShreyasZare . Will do.
New version 5.4 is now available which should fix this issue. Please reopen this if issue persists.
Dns ???
The latest version of DnsServer occupies a lot of CPU.