Crash

[muhanadali100]

I ran it for a whole day. I was surprised today morning that it was turned off until I made a reboot for the server and went back to work.

2020-07-05.log

[ShreyasZare]

Thanks for the detailed info. The logs are mostly network issues and does not give any particular info about the crash.

Do let me know about the OS and the hardware config that is being used. I would also suggest to keep an eye on the stats like CPU and memory utilization and try to find out the stats when the service is crashing.

Also do check the system logs which will include details of why the process crashed.

I see that you are using 8.8.8.8 and 1.1.1.1 forwarders for the DNS server. I suggest that you try running the server without any forwarders as a recursive resolver and see if you get any better results.

[muhanadali100]

I used Ubuntu-Server 16.04.6 Hardware : Core i5-3470 4Core-4threads Ram 8GB 1600 SSD 128GB

I used ESXI 6.5 log 5/7/2020 syslog.2.gz log 4/7/2020 syslog.3.gz

[ShreyasZare]

Thanks for the logs. The logs is showing that the dns-service is starting and stopping. I could not find any entry which could indicate reason for the service to crash. The systemd service config however has a restart option set to 10 seconds and if the service process crashes, it should start automatically.

How much RAM does the VM running on the ESXi has?

[muhanadali100]

ESXI 16GB DNS SERVER 8GB

The ESXI Has working Good , I installed 1- SAS Radius 2- Elastic Voip 3- DNS Server

The proplem just in DNS Server, when it's crash the ubuntu-server is working good but DNS SERVER Stopped

if it's stopped i will capture a video for the proplem

[ShreyasZare]

Thanks for the details. The DNS service should ideally restart automatically by systemd and if thats not happening then it could be that the process is not really crashing cleanly. When this scenario repeats, do check for clues to see if the process is indeed not running, check the CPU and memory stats and check if the process is still bound to the TCP and UDP ports, etc.

[muhanadali100]

2020-07-07.log

syslog.gz

https://youtu.be/Tf96s-GbZmY

I captured a video for the proplem

[ShreyasZare]

Thanks for the video and the logs. From the logs it seems that you are getting no response from the Google and Cloudflare forwarders for huge number of minutes like easily for more than 30 mins. This can be either some network issue at your end or that you are hitting the threshold for number of queries from single IP address and are being rate-limited. This will cause Google or Cloudflare services to not respond to any queries from your IP address. Since you have a lot of users, it is bound to happen at peak hours. I would recommend that you remove the forwarders configured and let the server run its own recursive resolver and see if that helps.

Adding more forwarders wont help since the DNS server will send queries to all the forwarders and get the response from the first one that responds thus getting quick responses. So adding more forwarders will just increase the outbound traffic and wont do any kind of load balancing.

In such scenarios (network issues etc) the server will have a thousands of threads waiting for response from the forwarders and that will cause using up all the threads available in the thread pool. The web console uses the same thread pool and in such case wont respond to requests. This is a likely scenario that is occurring and the server is not really crashing which explains why there is nothing in the syslog.

I would also suggest that instead of reboot the server each time there is no response from the DNS web console, just check the stats on the server like checking for "netstat -nlptu" to see if the dotnet process is running and bound to the tcp and udp ports. Do check for CPU and memory utilization for the dotnet process. Keep an eye on the network bandwidth being used at peak hours and at normal hours. This will help in understanding the issue better.

Since, you have many users, do consider having 2 DNS servers running for redundancy and load balancing. This is useful with any software that you use since there can be issues with servers and software freezing or crashing. Having a couple of servers will help keeping things working well.

[muhanadali100]

I apologized that I closed it by mistake

[muhanadali100]

in the ubuntu server I use nameserver 127.0.0.1 if i removed the forwarders the network is stopped how can i fix it ???

[ShreyasZare]

in the ubuntu server I use nameserver 127.0.0.1

If you mean for the resolv.conf then this is not used by the DNS server. The resolv.conf is just to let the system resolve DNS using the name servers.

if i removed the forwarders the network is stopped how can i fix it ???

If you remove the forwarders from the DNS server settings then no domain is resolved? If that is the case then its network issue or probably your ISP blocking outbound DNS queries. You will need to test this to confirm. Use the DNS Client tab and try to query using "Recursive Query" and see if domain names are resolving.

[muhanadali100]

[ShreyasZare]

Ahh! It seems your ISP is hijacking your DNS requests and that's why "f.gtld-servers.net" is responding with an "answer" to the www.facebook.com query. Probably, the domain names are being blocked by your ISP.

[muhanadali100]

I'm the isp , I have a puplic ip and I don't blocking anything , I made a dstnat address 15.15.15.4(DNS) to the 8.8.8.8 (Google) i disable it now

i check with netstat -nlptu

[muhanadali100]

Ahh! It seems your ISP is hijacking your DNS requests and that's why "f.gtld-servers.net" is responding with an "answer" to the www.facebook.com query. Probably, the domain names are being blocked by your ISP.

what i can do to fix the proplem , What is it supposed to give me when I use DNS Client ??

[ShreyasZare]

I'm the isp , I have a puplic ip and I don't blocking anything , I made a dstnat address 15.15.15.4(DNS) to the 8.8.8.8 (Google) i disable it now

Its not good to use public IP addresses that you are not assigned for internal private usage.

[ShreyasZare]

Ahh! It seems your ISP is hijacking your DNS requests and that's why "f.gtld-servers.net" is responding with an "answer" to the www.facebook.com query. Probably, the domain names are being blocked by your ISP.

what i can do to fix the proplem , What is it supposed to give me when I use DNS Client ??

Check this dnsclient.net link where you will see the response for a recursive query. You will see the the Facebook's name servers are responding with the answer whereas in your case, the Verisign name servers are responding with answer which means that someone on your network path is spoofing all IP addresses for "gtld-servers.net" and then responding with "answers" that could be filtered or modified.

[muhanadali100]

Ahh! It seems your ISP is hijacking your DNS requests and that's why "f.gtld-servers.net" is responding with an "answer" to the www.facebook.com query. Probably, the domain names are being blocked by your ISP.

what i can do to fix the proplem , What is it supposed to give me when I use DNS Client ??

Check this dnsclient.net link where you will see the response for a recursive query. You will see the the Facebook's name servers are responding with the answer whereas in your case, the Verisign name servers are responding with answer which means that someone on your network path is spoofing all IP addresses for "gtld-servers.net" and then responding with "answers" that could be filtered or modified.

can i make a drop for that ???

now my dns give me that

[ShreyasZare]

Ahh! It seems your ISP is hijacking your DNS requests and that's why "f.gtld-servers.net" is responding with an "answer" to the www.facebook.com query. Probably, the domain names are being blocked by your ISP.

what i can do to fix the proplem , What is it supposed to give me when I use DNS Client ??

Check this dnsclient.net link where you will see the response for a recursive query. You will see the the Facebook's name servers are responding with the answer whereas in your case, the Verisign name servers are responding with answer which means that someone on your network path is spoofing all IP addresses for "gtld-servers.net" and then responding with "answers" that could be filtered or modified.

can i make a drop for that ???

now my dns give me that

I am not sure what you are doing. Is that something you had configured?

[muhanadali100]

Ahh! It seems your ISP is hijacking your DNS requests and that's why "f.gtld-servers.net" is responding with an "answer" to the www.facebook.com query. Probably, the domain names are being blocked by your ISP.

what i can do to fix the proplem , What is it supposed to give me when I use DNS Client ??

Check this dnsclient.net link where you will see the response for a recursive query. You will see the the Facebook's name servers are responding with the answer whereas in your case, the Verisign name servers are responding with answer which means that someone on your network path is spoofing all IP addresses for "gtld-servers.net" and then responding with "answers" that could be filtered or modified.

can i make a drop for that ??? now my dns give me that

I am not sure what you are doing. Is that something you had configured?

Iam configured the ISP Server it's Mikrotik 1036 if anything i can do for best result of dns please tell me

[ShreyasZare]

Thanks for the inputs. I think I have some ideas for improving the DNS server to handle thread pool exhaustion properly. Will try to address this in upcoming updates.

[muhanadali100]

Can you tell me what the Hardware i need for 100 users or more please ,

[ShreyasZare]

Its not possible to give hardware requirements for any kind of application. It totally depends on the total number of requests the users generate. Any estimation requires trial and error initially to understand load for your use case.

However, I am adding Async IO into the DNS code which should improve the performance such that the server will handle more traffic with less threads.

[muhanadali100]

I install another system but it's same proplem it's crash

[ShreyasZare]

Thanks for the details. I am already working on a fix for this and will soon start testing.

[ShreyasZare]

New version is now available which implements async IO to prevent the server from stalling. Do update and see if the issue was fixed.