Weird DOH / QUIC issue - Githubissues

zs311521 commented 1 month ago

Hi,

Something weird I have noticed. I have my own DNS resolver. I got the technitium splash page, and then 10 mins later it turns into 404 Not Found.

This makes QUIC stop working, but also importantly DOH requests.

Please see sanitized logs

[2024-09-23 07:24:32 UTC] [0.0.0.0:853] [QUIC] System.Net.Quic.QuicException: Operation aborted. at System.Net.Quic.QuicListener.DisposeAsync() at System.Net.Quic.QuicListener.DisposeAsync() at DnsServerCore.Dns.DnsServer.StopAsync() in \DnsServerCore\Dns\DnsServer.cs:line 4965 at DnsServerCore.WebServiceSettingsApi.b__0() in \DnsServerCore\WebServiceSettingsApi.cs:line 170 at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread) --- End of stack trace --- at System.Net.Quic.QuicListener.AcceptConnectionAsync(CancellationToken cancellationToken) at DnsServerCore.Dns.DnsServer.AcceptQuicConnectionAsync(QuicListener quicListener) in \DnsServerCore\Dns\DnsServer.cs:line 785

[2024-09-23 07:24:32 UTC] [[::]:853] [QUIC] System.Net.Quic.QuicException: Operation aborted. at System.Net.Quic.QuicListener.DisposeAsync() at System.Net.Quic.QuicListener.DisposeAsync() at DnsServerCore.Dns.DnsServer.StopAsync() in \DnsServerCore\Dns\DnsServer.cs:line 4965 at DnsServerCore.WebServiceSettingsApi.b__0() in \DnsServerCore\WebServiceSettingsApi.cs:line 170 at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread) --- End of stack trace --- at System.Net.Quic.QuicListener.AcceptConnectionAsync(CancellationToken cancellationToken) at DnsServerCore.Dns.DnsServer.AcceptQuicConnectionAsync(QuicListener quicListener) in \DnsServerCore\Dns\DnsServer.cs:line 785

[2024-09-23 07:24:32 UTC] [[::]:443] [HTTPS] DNS Server was bound successfully. [2024-09-23 07:24:32 UTC] DNS service was restarted successfully. [2024-09-23 07:24:36 UTC] [] Check for update was done {updateAvailable: False; updateVersion: 13.0; updateTitle: New Update (v13.0) Available!; updateMessage: Follow the instructions from the link below to update the DNS server to the latest version. Read the change logs before installing this update to know if there are any breaking changes.; instructionsLink: https://blog.technitium.com/2017/11/running-dns-server-on-ubuntu-linux.html; changeLogLink: https://github.com/TechnitiumSoftware/DnsServer/blob/master/CHANGELOG.md;}

[2024-09-23 07:24:42 UTC] DNS Server config file was saved: /etc/dns/dns.config [2024-09-23 07:25:21 UTC] [] Check for update was done {updateAvailable: False; updateVersion: 13.0; updateTitle: New Update (v13.0) Available!; updateMessage: Follow the instructions from the link below to update the DNS server to the latest version. Read the change logs before installing this update to know if there are any breaking changes.; instructionsLink: https://blog.technitium.com/2017/11/running-dns-server-on-ubuntu-linux.html; changeLogLink: https://github.com/TechnitiumSoftware/DnsServer/blob/master/CHANGELOG.md;}

[2024-09-23 07:25:33 UTC] [] Check for update was done {updateAvailable: False; updateVersion: 13.0; updateTitle: New Update (v13.0) Available!; updateMessage: Follow the instructions from the link below to update the DNS server to the latest version. Read the change logs before installing this update to know if there are any breaking changes.; instructionsLink: https://blog.technitium.com/2017/11/running-dns-server-on-ubuntu-linux.html; changeLogLink: https://github.com/TechnitiumSoftware/DnsServer/blob/master/CHANGELOG.md;}

[2024-09-23 07:26:29 UTC] [] Check for update was done {updateAvailable: False; updateVersion: 13.0; updateTitle: New Update (v13.0) Available!; updateMessage: Follow the instructions from the link below to update the DNS server to the latest version. Read the change logs before installing this update to know if there are any breaking changes.; instructionsLink: https://blog.technitium.com/2017/11/running-dns-server-on-ubuntu-linux.html; changeLogLink: https://github.com/TechnitiumSoftware/DnsServer/blob/master/CHANGELOG.md;}

[2024-09-23 07:28:07 UTC] [] Check for update was done {updateAvailable: False; updateVersion: 13.0; updateTitle: New Update (v13.0) Available!; updateMessage: Follow the instructions from the link below to update the DNS server to the latest version. Read the change logs before installing this update to know if there are any breaking changes.; instructionsLink: https://blog.technitium.com/2017/11/running-dns-server-on-ubuntu-linux.html; changeLogLink: https://github.com/TechnitiumSoftware/DnsServer/blob/master/CHANGELOG.md;}

[2024-09-23 07:28:09 UTC] DNS Server failed to resolve the request '. HTTPS IN'. TechnitiumLibrary.Net.Dns.DnsClientResponseDnssecValidationException: DNSSEC validation failed due to invalid signature [DnssecBogus] for owner name: /A at TechnitiumLibrary.Net.Dns.DnsClient.DnssecValidateSignatureAsync(DnsDatagram response, IReadOnlyList1 records, IReadOnlyList1 dnsKeyRecords, IReadOnlyList`1 unsignedZones, DnssecValidateSignatureParameters parameters, Boolean isAuthoritySection, Boolean isAdditionalSection) in \TechnitiumLibrary.Net\Dns\DnsClient.cs:line 3048 --- End of stack trace ---

[2024-09-23 07:28:09 UTC] DNS Server failed to resolve the request '. A IN'. TechnitiumLibrary.Net.Dns.DnsClientResponseDnssecValidationException: DNSSEC validation failed due to invalid signature [DnssecBogus] for owner name: /A at TechnitiumLibrary.Net.Dns.DnsClient.DnssecValidateSignatureAsync(DnsDatagram response, IReadOnlyList1 records, IReadOnlyList1 dnsKeyRecords, IReadOnlyList`1 unsignedZones, DnssecValidateSignatureParameters parameters, Boolean isAuthoritySection, Boolean isAdditionalSection) in \TechnitiumLibrary.Net\Dns\DnsClient.cs:line 3048 --- End of stack trace ---

[2024-09-23 07:28:09 UTC] DNS Server failed to resolve the request '. AAAA IN'. TechnitiumLibrary.Net.Dns.DnsClientResponseDnssecValidationException: DNSSEC validation failed due to invalid signature [DnssecBogus] for

System.Net.Http.HttpRequestException: Connection refused (webdns.com:443) ---> System.Net.Sockets.SocketException (111): Connection refused at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) at System.Net.Sockets.Socket.gWaitForConnectWithCancellation|285_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken) at TechnitiumLibrary.Net.Dns.ClientConnection.HttpsClientConnection.ConnectCallback(SocketsHttpConnectionContext context, CancellationToken cancellationToken) in \TechnitiumLibrary.Net\Dns\ClientConnection\HttpsClientConnection.cs:line 153 at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken) --- End of inner exception stack trace --- at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.AddHttp2ConnectionAsync(QueueItem queueItem) at System.Threading.Tasks.TaskCompletionSourceWithCancellation`1.WaitWithCancellationAsync(CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) at TechnitiumLibrary.Net.Dns.ClientConnection.HttpsClientConnection.QueryAsync(DnsDatagram request, Int32 timeout, Int32 retries, CancellationToken cancellationToken) in \TechnitiumLibrary.Net\Dns\ClientConnection\HttpsClientConnection.cs:line 291 at TechnitiumLibrary.Net.Dns.DnsClient.<>cDisplayClass93_0.<g__DoResolveAsync|1>d.MoveNext() in \TechnitiumLibrary.Net\Dns\DnsClient.cs:line 4500 --- End of stack trace from previous location --- at TechnitiumLibrary.Net.Dns.DnsClient.<

ShreyasZare commented 1 month ago

Thanks for the feedback. It seems that the Quic error is coming after you clicked Save Settings on the panel. The QUIC library is throwing this exception causing the server to fail to restart the services correctly.

Try to restart the DNS server manually once and it should reload the saved setting and should work well. I am adding a check for this issue in the next update that is planned soon.

The other DoH errors listed are due to DNSSEC validation failures. And the last error is due to upstream not answering: System.Net.Http.HttpRequestException: Connection refused (<redacted>:443)

zs311521 commented 1 month ago

Thank you. I have DNSSEC configured so I’m not sure why it’s throwing an error causing DOH via HTTP/3 not work? Is it because my web server is using HTTP3? It’s strange as it causes a 404 not found randomly, I think until the web server is restarted.

ShreyasZare commented 1 month ago

Thank you. I have DNSSEC configured so I’m not sure why it’s throwing an error causing DOH via HTTP/3 not work? Is it because my web server is using HTTP3? It’s strange as it causes a 404 not found randomly, I think until the web server is restarted.

Please try to restart the DNS server and let me know if its working.

zs311521 commented 1 month ago

Yes, a restart fixes QUIC!

But the 404 is still on the root server, rather than the link to DOH Technitium page (not sure if this is related).

A restart also fixes DOH, BUT Only 3 requests come through and then there's a 404 error and no more forwards into the rescurive DNS. I swapped it back to QUIC and it works again.

9920 2024-09-23 18:11:05 14.203.91.190 Https Recursive NoError guzzoni-apple-com.v.aaplimg.com HTTPS IN
9919 2024-09-23 18:11:05 14.203.91.190 Https Recursive NoError guzzoni-apple-com.v.aaplimg.com AAAA IN
9918 2024-09-23 18:11:05 14.203.91.190 Https Recursive NoError guzzoni-apple-com.v.aaplimg.com A IN 3.105.196.9 9917 2024-09-23 18:11:02 14.203.91.190 Quic Recursive NxDomain nxdomain-2pi4cb8klzp.xyz HTTPS IN
9916 2024-09-23 18:11:02 14.203.91.190 Quic Recursive NxDomain nxdomain-2pi4cb8klzp.xyz A IN
9915 2024-09-23 18:11:02 14.203.91.190 Quic Recursive NxDomain nxdomain-2pi4cb8klzp.xyz AAAA IN

zs311521 commented 1 month ago

Spoke too soon, systemctl restart dns no longer fixes it.

It's interesting, I switch to HTTPS and it does queries for a second and just drops.

Flicking to QUIC forwards nothing.

TLS is like the perfect back-up, switch DOT back on and everything works perfectly.

I'd like to do HTTPS/3 or DOQ though! But I cannot pinpoint the issue :(

zs311521 commented 1 month ago

Please see further logs as soon as I enabeld forwardign to DOH on my Local Technitium. Noting it made about 5-8 successful queries before erroring (via the Cloud Resolver DNS logs)

System.Net.Http.HttpRequestException: Connection refused (:443) ---> System.Net.Sockets.SocketException (111): Connection refused at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) at System.Net.Sockets.Socket.<ConnectAsync>g__WaitForConnectWithCancellation|285_0(AwaitableSocketAsyncEventArgs saea, ValueTask connectTask, CancellationToken cancellationToken) at TechnitiumLibrary.Net.Dns.ClientConnection.HttpsClientConnection.ConnectCallback(SocketsHttpConnectionContext context, CancellationToken cancellationToken) in Z:\Technitium\Projects\TechnitiumLibrary\TechnitiumLibrary.Net\Dns\ClientConnection\HttpsClientConnection.cs:line 153 at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken) --- End of inner exception stack trace --- at System.Net.Http.HttpConnectionPool.ConnectToTcpHostAsync(String host, Int32 port, HttpRequestMessage initialRequest, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.AddHttp2ConnectionAsync(QueueItem queueItem) at System.Threading.Tasks.TaskCompletionSourceWithCancellation1.WaitWithCancellationAsync(CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpClient.gCore|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) at TechnitiumLibrary.Net.Dns.ClientConnection.HttpsClientConnection.QueryAsync(DnsDatagram request, Int32 timeout, Int32 retries, CancellationToken cancellationToken) in Z:\Technitium\Projects\TechnitiumLibrary\TechnitiumLibrary.Net\Dns\ClientConnection\HttpsClientConnection.cs:line 291 at TechnitiumLibrary.Net.Dns.DnsClient.<>c__DisplayClass93_0.<gDoResolveAsync|1>d.MoveNext() in Z:\Technitium\Projects\TechnitiumLibrary\TechnitiumLibrary.Net\Dns\DnsClient.cs:line 4500 --- End of stack trace from previous location --- at TechnitiumLibrary.Net.Dns.DnsClient.<

[2024-09-23 09:22:01 UTC] DNS Server config file was saved: /etc/dns/dns.config [2024-09-23 09:22:08 UTC] DNS Server failed to resolve the request 'nest-camera-media-upload.googleapis.com. A IN' using forwarders: https:///dns-query. TechnitiumLibrary.Net.Dns.DnsClientNoResponseException: DnsClient failed to resolve the request 'nest-camera-media-upload.googleapis.com. A IN': request timed out for name server [: (IP sanitized)]. at TechnitiumLibrary.Net.Dns.ClientConnection.QuicClientConnection.QueryAsync(DnsDatagram request, Int32 timeout, Int32 retries, CancellationToken cancellationToken) in Z:\Technitium\Projects\TechnitiumLibrary\TechnitiumLibrary.Net\Dns\ClientConnection\QuicClientConnection.cs:line 379 at TechnitiumLibrary.Net.Dns.DnsClient.<>cDisplayClass93_0.<gDoResolveAsync|1>d.MoveNext() in Z:\Technitium\Projects\TechnitiumLibrary\TechnitiumLibrary.Net\Dns\DnsClient.cs:line 4500 --- End of stack trace from previous location --- at TechnitiumLibrary.Net.Dns.DnsClient.<>cDisplayClass93_0.<gDoResolveAsync|1>d.MoveNext() in Z:\Technitium\Projects\TechnitiumLibrary\TechnitiumLibrary.Net\Dns\DnsClient.cs:line 4716 --- End of stack trace from previous location --- at TechnitiumLibrary.Net.Dns.DnsClient.<>cDisplayClass93_0.<gDoResolveAsync|1>d.MoveNext() in Z:\Technitium\Projects\TechnitiumLibrary\TechnitiumLibrary.Net\Dns\DnsClient.cs:line 4419 --- End of stack trace from previous location --- at TechnitiumLibrary.Net.Dns.DnsClient.InternalResolveAsync(DnsDatagram request, Func`3 getValidatedResponseAsync, Boolean doNotReorderNameServers, CancellationToken cancellationToken) in Z:\Technitium\Projects\TechnitiumLibrary\TechnitiumLibrary.Net\Dns\DnsClient.cs:line 4841 at TechnitiumLibrary.Net.Dns.DnsClient.InternalDnssecResolveAsync(DnsQuestionRecord question, CancellationToken cancellationToken) in Z:\Technitium\Projects\TechnitiumLibrary\TechnitiumLibrary.Net\Dns\DnsClient.cs:line 4894 at TechnitiumLibrary.Net.Dns.DnsClient.<>cDisplayClass97_0.<b0>d.MoveNext() in Z:\Technitium\Projects\TechnitiumLibrary\TechnitiumLibrary.Net\Dns\DnsClient.cs:line 4993 --- End of stack trace from previous location --- at TechnitiumLibrary.Net.Dns.DnsClient.ResolveQueryAsync(DnsQuestionRecord question, Func2 resolveAsync) in Z:\Technitium\Projects\TechnitiumLibrary\TechnitiumLibrary.Net\Dns\DnsClient.cs:line 4254 at TechnitiumLibrary.Net.Dns.DnsClient.InternalCachedResolveQueryAsync(DnsQuestionRecord question, CancellationToken cancellationToken) in Z:\Technitium\Projects\TechnitiumLibrary\TechnitiumLibrary.Net\Dns\DnsClient.cs:line 4975 at DnsServerCore.Dns.DnsServer.DefaultRecursiveResolveAsync(DnsQuestionRecord question, NetworkAddress eDnsClientSubnet, IDnsCache dnsCache, Boolean dnssecValidation, Boolean skipDnsAppAuthoritativeRequestHandlers, CancellationToken cancellationToken) in Z:\Technitium\Projects\DnsServer\DnsServerCore\Dns\DnsServer.cs:line 3346 at DnsServerCore.Dns.DnsServer.RecursiveResolverBackgroundTaskAsync(DnsQuestionRecord question, NetworkAddress eDnsClientSubnet, Boolean advancedForwardingClientSubnet, IReadOnlyList1 conditionalForwarders, Boolean dnssecValidation, Boolean cachePrefetchOperation, Boolean cacheRefreshOperation, Boolean skipDnsAppAuthoritativeRequestHandlers, TaskCompletionSource1 taskCompletionSource) in Z:\Technitium\Projects\DnsServer\DnsServerCore\Dns\DnsServer.cs:line 3133 [2024-09-23 09:22:08 UTC] DNS Server failed to resolve the request 'nest-camera-media-upload.googleapis.com. AAAA IN' using forwarders: https://<sanitized>/dns-query. TechnitiumLibrary.Net.Dns.DnsClientNoResponseException: DnsClient failed to resolve the request 'nest-camera-media-upload.googleapis.com. AAAA IN': request timed out for name server [: (IP sanitized)].

ShreyasZare commented 1 month ago

Just to be sure, you had two DNS server setup, one on a VPS that provides DoH/DoQ/DoT services and one you use locally, right?

If yes, then these logs you shared seems to be of local DNS server. You should check the logs on the upstream DNS server to know why it stopped the service. Local logs are just saying that its either timeout issue or connection refused.

zs311521 commented 1 month ago

That's correct, local DNS server and a VPS with Technitium installed that has authoritative and recursive DNS.

I've configured it to only accept queries from my ipv4/ipv6 address, and opened up the relevant ports (except HTTP port 80 as as I enabled RFC DNS root cert updates). I also turned off HTTP from optional settings in the upstream server, but everything else on in. The web GUI server is accessible via https and http/3 (I've edited the apache config file accordingly to support http/3 in the GUI)

Weirdly, the logs do not show anything on my VPS server ('view logs).

On the VPS server, for my public IPv4, it shows a successful DOH query or 3, and then immediately all queries stop. like when QUIC randomly works, it shows QUIC queries and jsut stops. I flick forwarding back to DOT and it works grea.t

ShreyasZare commented 1 month ago

Thanks for clarifying. It seems to be the similar issue you have described in #1030 regarding QUIC.

Since you have Apache for reverse proxying to admin panel, are you also using the same for DoH service?

zs311521 commented 1 month ago

Sorry my mistake, it's via nginx

Here is my nginx conf file ( /etc/nginx/nginx.conf) - this is all of hte extra config I've done - everything else is OOB.

That's all the config (below)

` user nginx; worker_processes auto;

error_log /var/log/nginx/error.log notice; pid /var/run/nginx.pid;

events { worker_connections 1024; }

server { listen 80; listen [::]:80; server_name my server.com www.myserver.com;

# Redirect all HTTP traffic to HTTPS
location / {
    return 301 https://$server_name$request_uri;
}

}

http { include /etc/nginx/mime.types; default_type application/octet-stream;

log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                  '$status $body_bytes_sent "$http_referer" '
                  '"$http_user_agent" "$http_x_forwarded_for"';

access_log  /var/log/nginx/access.log  main;

sendfile        on;
#tcp_nopush     on;

server { listen 443 quic reuseport; # Enable QUIC and HTTP/3 over UDP listen [::]:443 quic reuseport; # Enable QUIC for IPv6 over UDP server_name myserver.com;

ssl_certificate /etc/letsencrypt/live/myserver.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/myserver.com/privkey.pem;
ssl_protocols TLSv1.3;              # HTTP/3 requires TLS 1.3

# SSL settings for QUIC/HTTP/3
ssl_prefer_server_ciphers off;
ssl_session_timeout 1d;
ssl_session_cache shared:SSL:10m;

# Add the Alt-Svc header to advertise HTTP/3 support
add_header Alt-Svc 'h3-23=":443"; ma=86400';  # Advertise HTTP/3

add_header Alt-Svc 'h3-23=":443"; ma=86400'; # Advertise HTTP/3 support on por>

location / {
    # Define location if needed
    try_files $uri $uri/ =404;
}

} `

zs311521 commented 1 month ago

Ok, so after a bit of troubleshooting. 6 queries over DOH3 successful, and then seconds later it just 'dies'. See below:

`root@localhost:~# sudo tail -f /var/log/nginx/access.log MYIP - - [23/Sep/2024:13:03:17 +0000] "POST /dns-query HTTP/3.0" 500 0 "-" "DoH client" "-" MYIP- - [23/Sep/2024:13:03:17 +0000] "POST /dns-query HTTP/3.0" 500 0 "-" "DoH client" "-" MYIP- - [23/Sep/2024:13:03:17 +0000] "POST /dns-query HTTP/3.0" 500 0 "-" "DoH client" "-" MYIP - - [23/Sep/2024:13:03:17 +0000] "POST /dns-query HTTP/3.0" 500 0 "-" "DoH client" "-" MYIP - - [23/Sep/2024:13:03:17 +0000] "POST /dns-query HTTP/3.0" 500 0 "-" "DoH client" "-" MYIP- - [23/Sep/2024:13:03:17 +0000] "POST /dns-query HTTP/3.0" 500 0 "-" "DoH client" "-" MYIP- - [23/Sep/2024:13:03:17 +0000] "POST /dns-query HTTP/3.0" 500 0 "-" "DoH client" "-"

No errors in sudo tail -f /var/log/nginx/error.log

2024/09/23 13:02:55 [notice] 52990#52990: worker process 52991 exited with code 0 2024/09/23 13:02:55 [notice] 52990#52990: exit 2024/09/23 13:02:55 [notice] 53019#53019: using the "epoll" event method 2024/09/23 13:02:55 [notice] 53019#53019: nginx/1.26.2 2024/09/23 13:02:55 [notice] 53019#53019: built by gcc 13.2.0 (Ubuntu 13.2.0-23ubuntu4) 2024/09/23 13:02:55 [notice] 53019#53019: OS: Linux 6.10.2-x86_64-linode165 2024/09/23 13:02:55 [notice] 53019#53019: getrlimit(RLIMIT_NOFILE): 1024:524288 2024/09/23 13:02:55 [notice] 53020#53020: start worker processes 2024/09/23 13:02:55 [notice] 53020#53020: start worker process 53021 2024/09/23 13:02:55 [notice] 53020#53020: start worker process 53022

My nginx.conf file:

server {
    listen 443 quic reuseport;      # Enable QUIC and HTTP/3 over UDP
    listen [::]:443 quic reuseport; # Enable QUIC for IPv6 over UDP
    server_name mysite.com;

    ssl_certificate /etc/letsencrypt/live/mysite.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/mysite.com/privkey.pem;
    ssl_protocols TLSv1.3;              # HTTP/3 requires TLS 1.3

    # SSL settings for QUIC/HTTP/3
    ssl_prefer_server_ciphers off;
    ssl_session_timeout 1d;
    ssl_session_cache shared:SSL:10m;

    # Add the Alt-Svc header to advertise HTTP/3 support
    add_header Alt-Svc 'h3-23=":443"; ma=86400';  # Advertise HTTP/3 support

 # Handle DNS over HTTPS requests
    location /dns-query {
proxy_pass https://mywebsite.com:53443/dns-query;
proxy_ssl_trusted_certificate /etc/ssl/certs/ca-certificates.crt;
proxy_ssl_verify on;
proxy_ssl_verify_depth 2;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;

proxy_read_timeout 300; proxy_connect_timeout 300; proxy_send_timeout 300; proxy_buffering off;

proxy_buffering off;

} }

ShreyasZare commented 1 month ago

Thanks for the details. If you are using nginx to reverse proxy to DoH service then you should enable the DNS-over-HTTP optional protocol option instead and use it in your proxy_pass directive. You do not need to have another TLS layer since your nginx is already handling TLS.

Another thing is that I believe you need to set the timeout values correctly by specifying s as the seconds unit. There is currently no unit configured so I think its being read as milliseconds and this may be the cause of this issue. So the directive should be like below:

proxy_read_timeout 300s;

Same with the other directives where you specify timeout values.

Other than that, I am not sure why the request is failing after few attempts. I would suggest that you use the Optional Protocols directly instead of reverse proxying via nginx and see if it works.

ShreyasZare commented 1 month ago

Technitium DNS Server v13.0.1 is now available that will handle the QUIC error you had earlier. Do update and let me know your feedback.

zs311521 commented 1 month ago

Technitium DNS Server v13.0.1 is now available that will handle the QUIC error you had earlier. Do update and let me know your feedback.

Thank you, I will try the optional protocols directly.

I've just updated and am testing the QUIC connections as we speak. Looks good so far - I'm curious to see what the issue was? I feel like it will end up timing out again in 10-15 minutes, and is probably related to my issue with DoH on my server.

I will let you know though, but I'm probably not the best test subject for QUIC and DOH issues!

zs311521 commented 1 month ago

Sadly back to the errors. Nothing in resolver server logs, this is local

4718 --- End of stack trace from previous location --- at TechnitiumLibrary.Net.Dns.DnsClient.<>cDisplayClass93_0.<gDoResolveAsync|1>d.MoveNext() in Z:\Technitium\Projects\TechnitiumLibrary\TechnitiumLibrary.Net\Dns\DnsClient.cs:line 4419 --- End of stack trace from previous location --- at TechnitiumLibrary.Net.Dns.DnsClient.InternalResolveAsync(DnsDatagram request, Func`3 getValidatedResponseAsync, Boolean doNotReorderNameServers, CancellationToken cancellationToken) in Z:\Technitium\Projects\TechnitiumLibrary\TechnitiumLibrary.Net\Dns\DnsClient.cs:line 4843 at TechnitiumLibrary.Net.Dns.DnsClient.InternalDnssecResolveAsync(DnsQuestionRecord question, CancellationToken cancellationToken) in Z:\Technitium\Projects\TechnitiumLibrary\TechnitiumLibrary.Net\Dns\DnsClient.cs:line 4896 at TechnitiumLibrary.Net.Dns.DnsClient.<>cDisplayClass97_0.<b0>d.MoveNext() in Z:\Technitium\Projects\TechnitiumLibrary\TechnitiumLibrary.Net\Dns\DnsClient.cs:line 5000 --- End of stack trace from previous location --- at TechnitiumLibrary.Net.Dns.DnsClient.ResolveQueryAsync(DnsQuestionRecord question, Func2 resolveAsync) in Z:\Technitium\Projects\TechnitiumLibrary\TechnitiumLibrary.Net\Dns\DnsClient.cs:line 4254 at TechnitiumLibrary.Net.Dns.DnsClient.InternalCachedResolveQueryAsync(DnsQuestionRecord question, CancellationToken cancellationToken) in Z:\Technitium\Projects\TechnitiumLibrary\TechnitiumLibrary.Net\Dns\DnsClient.cs:line 4977 at DnsServerCore.Dns.DnsServer.DefaultRecursiveResolveAsync(DnsQuestionRecord question, NetworkAddress eDnsClientSubnet, IDnsCache dnsCache, Boolean dnssecValidation, Boolean skipDnsAppAuthoritativeRequestHandlers, CancellationToken cancellationToken) in Z:\Technitium\Projects\DnsServer\DnsServerCore\Dns\DnsServer.cs:line 3349 at DnsServerCore.Dns.DnsServer.RecursiveResolverBackgroundTaskAsync(DnsQuestionRecord question, NetworkAddress eDnsClientSubnet, Boolean advancedForwardingClientSubnet, IReadOnlyList1 conditionalForwarders, Boolean dnssecValidation, Boolean cachePrefetchOperation, Boolean cacheRefreshOperation, Boolean skipDnsAppAuthoritativeRequestHandlers, TaskCompletionSource1 taskCompletionSource) in Z:\Technitium\Projects\DnsServer\DnsServerCore\Dns\DnsServer.cs:line 3133 [2024-09-23 14:36:04 UTC] DNS Server failed to resolve the request 'ips1.unifi-ai.com. AAAA IN' using forwarders: [REMOVED]. TechnitiumLibrary.Net.Dns.DnsClientNoResponseException: DnsClient failed to resolve the request 'ips1.unifi-ai.com. AAAA IN': request timed out for name server [REMOVED]. at TechnitiumLibrary.Net.Dns.ClientConnection.QuicClientConnection.QueryAsync(DnsDatagram request, Int32 timeout, Int32 retries, CancellationToken cancellationToken) in Z:\Technitium\Projects\TechnitiumLibrary\TechnitiumLibrary.Net\Dns\ClientConnection\QuicClientConnection.cs:line 379 at TechnitiumLibrary.Net.Dns.DnsClient.<>c__DisplayClass93_0.<<InternalResolveAsync>g__DoResolveAsync|1>d.MoveNext() in Z:\Technitium\Projects\TechnitiumLibrary\TechnitiumLibrary.Net\Dns\DnsClient.cs:line 4502 --- End of stack trace from previous location --- at TechnitiumLibrary.Net.Dns.DnsClient.<>c__DisplayClass93_0.<<InternalResolveAsync>g__DoResolveAsync|1>d.MoveNext() in Z:\Technitium\Projects\TechnitiumLibrary\TechnitiumLibrary.Net\Dns\DnsClient.cs:line 4718 --- End of stack trace from previous location --- at TechnitiumLibrary.Net.Dns.DnsClient.<>c__DisplayClass93_0.<<InternalResolveAsync>g__DoResolveAsync|1>d.MoveNext() in Z:\Technitium\Projects\TechnitiumLibrary\TechnitiumLibrary.Net\Dns\DnsClient.cs:line 4419 --- End of stack trace from previous location --- at TechnitiumLibrary.Net.Dns.DnsClient.InternalResolveAsync(DnsDatagram request, Func3 getValidatedResponseAsync, Boolean doNotReorderNameServers, CancellationToken cancellationToken) in Z:\Technitium\Projects\TechnitiumLibrary\TechnitiumLibrary.Net\Dns\DnsClient.cs:line 4843 at TechnitiumLibrary.Net.Dns.DnsClient.InternalDnssecResolveAsync(DnsQuestionRecord question, CancellationToken cancellationToken) in Z:\Technitium\Projects\TechnitiumLibrary\TechnitiumLibrary.Net\Dns\DnsClient.cs:line 4896 at TechnitiumLibrary.Net.Dns.DnsClient.<>c__DisplayClass97_0.<b__⬤

ShreyasZare commented 1 month ago

It seems that you have issues with the config since these errors are not reproducible otherwise. I would suggest that you take screenshots of all of your settings for both your upstream and local DNS servers and share it with support@technitium.com so as to get recommendations on it.

zs311521 commented 1 month ago

So I think my router isn't a fan of QUIC.

DOH works perfectly, but not when I set the h3 flag. When I set the H3 flag, and I check wireshark logs, I get:

1 0.000000000 HOME IP → SERVER IP tial, DCID=XXXXX, PKN: 0, CRYPTO, PADDING 2 0.041707079 SERVER IP → HOME IP QUIC 1262 Initial, SCID=XXXXX, PKN: 0, ACK, CC, PADDING 3 0.099365944 HOME IP → SERVER IP QUIC 1262 Initial, DCID=XXXXX, PKN: 0, CRYPTO, PADDING 4 0.137668324 SERVER IP → HOME IP QUIC 1262 Initial, SCID=XXXXX, PKN: 0, ACK, CC, PADDING

No other queries, it jsut gets stuck there. once I put the https flag back on, it queries via TLSv.1, I presume http1/2

HTTP/3 works perfectly on my server when I inspect via browser. I have NGINX disabled / turned off as it was causing issues.

Interestingly, when QUIC stops working, I see via the DNS Client I can ping QUIC - I cannot! I restart my router, and then I can ping QUIC again. I think I have finally narrowed it down to the router.

In terms of DOH, I finally got it workign (not HTTP/3). Turned out, my router IPS was silently blocking DOH requests, so I enabled the IP to/from server<>router and DOH works fine now.

It still doesnt like QUIC, and wondering whether I need to open firewall rules to stop it from randomly dropping.

When I do TSHARK, I successfully get 7 0.017921471 HOME IP → SERVER IP DTLS 83 Continuation Data So looks like QUIC is workign like a dream..... until it stops!

zs311521 commented 1 month ago

IPS/IDS looks like it may have been the issue for Unifi - unchecking 'DNS" and 'DOS' seems to have fixed it.

I narrowed down the issue of, QUIC connections stopped working and simply restarted a restart of the Unifi Router. I then unchecked those IPS/IDS settings and allowed INTERNET OUT of my Technitium IP as well as LOCAL (explicitly, to prevent rate limiting and other funky stuff UDM does).

Looks to have fixed the issue. BUT, not the DOH issue - which may be separate to this.

Phew! What an adventure.

zs311521 commented 1 month ago

Ok: So it looks like the fix was short-lived. QUIC stopped working again. I restarted the router. It still didnt work. I had to flick it from QUIC TO DOT, and then back to QUIC (upstream forwarding) and then queries went all-in again.

So I realised: A simple restart of my router doesnt automatically start queries via QUIC again. I need to restart, and then flick forwarding from QUIC to DOT, and then Back to QUIC again. and it works... for a time. This makes me think it wasn't JUST my settings, and could also be server-related??

zs311521 commented 1 month ago

Note: In my firewall rules, I flicked 'UDP other' to 180 from 30. That may do something re state timeouts? Let' see as well.

ShreyasZare commented 1 month ago

Thanks for posting details here so that it helps someone in same situation.

Ok: So it looks like the fix was short-lived. QUIC stopped working again. I restarted the router. It still didnt work. I had to flick it from QUIC TO DOT, and then back to QUIC (upstream forwarding) and then queries went all-in again.

So I realised: A simple restart of my router doesnt automatically start queries via QUIC again. I need to restart, and then flick forwarding from QUIC to DOT, and then Back to QUIC again. and it works... for a time. This makes me think it wasn't JUST my settings, and could also be server-related??

The DNS server uses a single QUIC connection and reuses it. It seems that the QUIC connection fails to realize that there is some issue and does not do a clean disconnect to begin a new session. This may be since QUIC session is supposed to last even when your IP changes and thus the library is probably handling it this way. So when you switch to DoT, the existing DoQ connection gets closed and when you switch back to it then it starts working with a fresh connection which works.

Note: In my firewall rules, I flicked 'UDP other' to 180 from 30. That may do something re state timeouts? Let' see as well.

I think the state timeout may also have an effect on QUIC since it would remove any entries from state table if there was no QUIC packet during this interval. Try increasing it to something like 300 and see if the issue stops occurring or if it works for longer.

If this issue still persists, I would suggest that you use DoT or DoH (without h3) as an alternative since those too will give similar performance which wont be noticeably any different.

zs311521 commented 1 month ago

Thanks for posting details here so that it helps someone in same situation.

Ok: So it looks like the fix was short-lived. QUIC stopped working again. I restarted the router. It still didnt work. I had to flick it from QUIC TO DOT, and then back to QUIC (upstream forwarding) and then queries went all-in again. So I realised: A simple restart of my router doesnt automatically start queries via QUIC again. I need to restart, and then flick forwarding from QUIC to DOT, and then Back to QUIC again. and it works... for a time. This makes me think it wasn't JUST my settings, and could also be server-related??

The DNS server uses a single QUIC connection and reuses it. It seems that the QUIC connection fails to realize that there is some issue and does not do a clean disconnect to begin a new session. This may be since QUIC session is supposed to last even when your IP changes and thus the library is probably handling it this way. So when you switch to DoT, the existing DoQ connection gets closed and when you switch back to it then it starts working with a fresh connection which works.

Note: In my firewall rules, I flicked 'UDP other' to 180 from 30. That may do something re state timeouts? Let' see as well.

I think the state timeout may also have an effect on QUIC since it would remove any entries from state table if there was no QUIC packet during this interval. Try increasing it to something like 300 and see if the issue stops occurring or if it works for longer.

If this issue still persists, I would suggest that you use DoT or DoH (without h3) as an alternative since those too will give similar performance which wont be noticeably any different.

Just an update on this one. I disabled everything except QUIC and I have had success with successful forwarding via QUIC exclusively for the past 8 hours, which is the longest it has got to. I am confident it will remain and I haven't seen any errors since.

Sounds like DOH may have been conflicting with the QUIC somehow, perhaps DOH3.

I haven't tried DOH/3 exclusively as I don't think it's as reliable as QUIC, and not really necessary as QUIC is the best service for speed and privacy (I think).

I also disabled DOT just to be safe but it probably didn't have any impact (even though they both use port 853, once uses TCP and one uses UDP so probably no conflicts here).

Exclusively QUIC works great. I may try mixing QUIC / H3 in the future once libsmquic is updated to be more stable, but for now I can happily use QUIC exclusively!!

ShreyasZare commented 1 month ago

Just an update on this one. I disabled everything except QUIC and I have had success with successful forwarding via QUIC exclusively for the past 8 hours, which is the longest it has got to. I am confident it will remain and I haven't seen any errors since.

Good to know that!

I haven't tried DOH/3 exclusively as I don't think it's as reliable as QUIC, and not really necessary as QUIC is the best service for speed and privacy (I think).

DoH/3 uses same QUIC protocol that DNS-over-QUIC also uses. So, both of them have same properties.

zs311521 commented 1 month ago

Just an update on this one. I disabled everything except QUIC and I have had success with successful forwarding via QUIC exclusively for the past 8 hours, which is the longest it has got to. I am confident it will remain and I haven't seen any errors since.

Good to know that!

I haven't tried DOH/3 exclusively as I don't think it's as reliable as QUIC, and not really necessary as QUIC is the best service for speed and privacy (I think).

DoH/3 uses same QUIC protocol that DNS-over-QUIC also uses. So, both of them have same properties.

So good news is that DOQ connections are persisting and no longer dropping, UNLESS the upstream server is restarted OR block lists are updated on my local DNS server.

When this happens, I need to swap from QUIC to DOH forwarding, and back to QUIC. The connection then persists. Swapping from QUIC to UDP doesn't work, it needs to be to one of the encrypted protocols.

Overall, I'm happy as DOQ no longer randomly drops and only drops when I intentionally update blocklist manually (add a new entry) or restart the server DNS.

But , in case this is not an issue specific to me, I hope this info helps.

zs311521 commented 5 days ago

@ShreyasZare

Wohoo! the h3 URL scheme now works flawlessly when before it only did an initial handshake.

Please see logs. I don't need DOQ - is it ok to disable DOQ and only enable DOH/3?

I noticed that when I restart the DNS, the following queries come in per the first block via Tshark. After a few minutes I run tshark again I get the following in the logs (second block). I presume both of these means QUIC is working, not sure why the second block doesnt say QUIC.

All in all, amazing work Shreyas. I was going crazy thinking it was a config on my end, but so happy that yo ufound the fix. I tested this for DOH/3. I have not yet tested QUIC but I am all but certain it will no longer randomly drop off. I will continue using DOH/3 and it work so well - thank you.

   42 0.226084142 [Source_IP - IPv6] → [Destination_IP - IPv6] QUIC 88 Protected Payload (KP0)
   43 0.227322128 [Source_IP - IPv6] → [Destination_IP - IPv6] QUIC 232 Protected Payload (KP0)
   44 0.227560969 [Source_IP - IPv6] → [Destination_IP - IPv6] QUIC 91 Protected Payload (KP0)
   45 0.228334473 [Source_IP - IPv6] → [Destination_IP - IPv6] QUIC 220 Protected Payload (KP0)
   46 0.228577444 [Source_IP - IPv6] → [Destination_IP - IPv6] QUIC 91 Protected Payload (KP0)
   47 0.230070722 [Destination_IP - IPv6] → [Source_IP - IPv6] QUIC 99 Protected Payload (KP0), DCID=82a3d8a2c3a31c0969
   48 0.230240982 [Source_IP - IPv6] → [Destination_IP - IPv6] QUIC 93 Protected Payload (KP0)
   49 0.231155367 [Source_IP - IPv6] → [Destination_IP - IPv6] QUIC 88 Protected Payload (KP0)
   50 0.234987677 [Destination_IP - IPv6] → [Source_IP - IPv6] QUIC 99 Protected Payload (KP0), DCID=82a3d8a2c3a31c0969
   51 0.244905018 [Destination_IP - IPv6] → [Source_IP - IPv6] QUIC 104 Protected Payload (KP0), DCID=82a3d8a2c3a31c0969
   52 0.245204040 [Destination_IP - IPv6] → [Source_IP - IPv6] QUIC 104 Protected Payload (KP0), DCID=82a3d8a2c3a31c0969
   53 0.245770193 [Source_IP - IPv6] → [Destination_IP - IPv6] QUIC 90 Protected Payload (KP0)
   54 0.246280665 [Source_IP - IPv6] → [Destination_IP - IPv6] QUIC 88 Protected Payload (KP0)
   55 0.280950045 [Destination_IP - IPv6] → [Source_IP - IPv6] QUIC 100 Protected Payload (KP0), DCID=82a3d8a2c3a31c0969

31 10.001681885 [Destination_IP - IPv6] → [Source_IP - IPv6] UDP 108 33331 → 443 Len=44 32 10.001681965 [Destination_IP - IPv6] → [Source_IP - IPv6] UDP 292 33331 → 443 Len=228 33 10.001682045 [Destination_IP - IPv6] → [Source_IP - IPv6] UDP 101 33331 → 443 Len=37 34 10.001901366 [Source_IP - IPv6] → [Destination_IP - IPv6] UDP 96 443 → 33331 Len=32 35 10.002870451 [Source_IP - IPv6] → [Destination_IP - IPv6] UDP 351 443 → 33331 Len=287 36 10.003344204 [Source_IP - IPv6] → [Destination_IP - IPv6] UDP 422 443 → 33331 Len=358 37 10.003766866 [Source_IP - IPv6] → [Destination_IP - IPv6] UDP 99 443 → 33331 Len=35 38 10.003986248 [Source_IP - IPv6] → [Destination_IP - IPv6] UDP 92 443 → 33331 Len=28 39 10.010922164 [Destination_IP - IPv6] → [Source_IP - IPv6] UDP 100 33331 → 443 Len=36 40 10.015289227 [Destination_IP - IPv6] → [Source_IP - IPv6] UDP 100 33331 → 443 Len=36 41 10.015423588 [Source_IP - IPv6] → [Destination_IP - IPv6] UDP 94 443 → 33331 Len=30 42 10.050154371 [Destination_IP - IPv6] → [Source_IP - IPv6] UDP 101 33331 → 443 Len=37

ShreyasZare commented 5 days ago

Thanks for the feedback! Tshark may not recognize its QUIC if capture starts mid stream so that seems to be the reason here.

ShreyasZare commented 1 day ago

Technitium DNS Server v13.2 is now available which fixes this issue. Do update and let me know your feedback.

TechnitiumSoftware / DnsServer

Weird DOH / QUIC issue #1041