AdGuard Home's DNS-over-QUIC server functionality fails to make itself available on LAN IPs

DandelionSprout commented 3 years ago

Have a question or an idea? Please search it on our forum to make sure it was not yet asked. If you cannot find what you had in mind, please submit it here.

Prerequisites

Please answer the following questions for yourself before submitting an issue. YOU MAY DELETE THE PREREQUISITES SECTION.

[x] I am running the latest version
[x] I checked the documentation and found no answer
[x] I checked to make sure that this issue has not already been filed

Issue Details

Version of AdGuard Home server:
- v0.107.0-a.136+550b1798
How did you install AdGuard Home:
- GitHub releases
How did you setup DNS configuration:
- System
If it's a router or IoT, please write device model:
- Raspberry Pi 4 8GB
CPU architecture:
- ARM64
Operating system and version:
- Fedora Workstation 34

Expected Behavior

AGH's DNS-over-QUIC server can be accessed from LAN home devices.

Actual Behavior

Okay, so I had for some weeks thought that my connection problems to quic://dandelionsprout.asuscomm.com:48582 were the result of failing to scan IPv4 addresses on IPv6-able connections. That may or may not have been the case, but now the problem has either changed in nature, or I was incorrect to begin with; since further testing tonight, led me to the conclusion that it's a problem with LAN connections instead:

(Easiest to reproduce with AdGuard for Android, as steps 5-7 hinge on a phone connection.) 1) Have an AdGuard Home server with DNS-over-QUIC functionality, and with port forwarding set up for it on the home's router. 2) Connect your phone to a Wi-Fi network in the same home. 3) Connect to your DNS-over-QUIC server at AdGuard Settings → DNS filtering → Choose DNS server → Add custom DNS server. 4) See the "No connection" error. 5) Disconnect the phone from the Wi-Fi network, and change to using a 4G connection. 6) Repeat step 3. 7) See that the DNS server is connected now.

Screenshots

Screenshot:

N/A at the time of writing.

Additional Information

Successor to https://github.com/AdguardTeam/AdguardForAndroid/issues/3927 and https://github.com/AdguardTeam/AdguardForWindows/issues/3878

AdGuard for Android log: adguard.log — Highlight sequence:

21:07:32.514 [AsyncTask #14] INFO  c.a.android.filtering.commons.d - Found IPv6 address for network name:wlan0 (wlan0)
21:07:37.664 [AsyncTask #14] WARN  com.adguard.android.filtering.dns.c - Failed to test quic://dandelionsprout.asuscomm.com:48582
java.lang.IllegalArgumentException: Request timed out
    at com.adguard.dnslibs.proxy.DnsProxy.testUpstream(DnsProxy.java:236)
    at com.adguard.android.filtering.dns.c.a(DnsProxyUtils.java:88)
    at com.adguard.android.filtering.dns.c.a(DnsProxyUtils.java:51)
    at com.adguard.android.ui.CustomDnsActivity$a.doInBackground(CustomDnsActivity.java:2314)
    at android.os.AsyncTask$3.call(AsyncTask.java:378)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at android.os.AsyncTask$SerialExecutor$1.run(AsyncTask.java:289)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
    at java.lang.Thread.run(Thread.java:919)

AdGuard for Windows log: service_18-08-2021-20_36_13.647-2021-08-18.log — Highlight sequence:

INFO, AdguardSvc, 38, 18.08.2021 20:39:40.757, Start creating DNS servers https://dandelionsprout.asuscomm.com:2501/dns-query
quic://dandelionsprout.asuscomm.com:48582
INFO, AdguardSvc, 38, 18.08.2021 20:39:40.757, Creating DNS servers has been completed successfully
INFO, AdguardSvc, 38, 18.08.2021 20:39:40.758, Start testing DNS server DNS server: , (https://dandelionsprout.asuscomm.com:2501/dns-query, quic://dandelionsprout.asuscomm.com:48582; 0), type: Multi
INFO, AdguardSvc, 38, 18.08.2021 20:39:40.812, Adguard.Dns.Api.DnsApi | Testing upstream
INFO, AdguardSvc, 38, 18.08.2021 20:39:40.812, Adguard.Dns.Api.DnsApi | Start testing upstream Adguard.Dns.Api.DnsProxyServer.Configs.UpstreamOptions
INFO, AdguardSvc, 38, 18.08.2021 20:39:40.977, Adguard.Dns.Logging.DnsLoggerAdapter | DOH upstream: Destroying...
INFO, AdguardSvc, 38, 18.08.2021 20:39:40.978, Adguard.Dns.Logging.DnsLoggerAdapter | DOH upstream: Stopping event loop...
INFO, AdguardSvc, 38, 18.08.2021 20:39:40.978, Adguard.Dns.Logging.DnsLoggerAdapter | DOH upstream: Done
INFO, AdguardSvc, 38, 18.08.2021 20:39:40.978, Adguard.Dns.Logging.DnsLoggerAdapter | DOH upstream: Waiting all requests completed...
INFO, AdguardSvc, 38, 18.08.2021 20:39:40.978, Adguard.Dns.Logging.DnsLoggerAdapter | DOH upstream: Done
INFO, AdguardSvc, 38, 18.08.2021 20:39:40.978, Adguard.Dns.Logging.DnsLoggerAdapter | DOH upstream: Destroyed
INFO, AdguardSvc, 38, 18.08.2021 20:39:40.978, Adguard.Dns.Api.DnsApi | Testing upstream has been completed successfully
INFO, AdguardSvc, 38, 18.08.2021 20:39:40.978, Adguard.Dns.Api.DnsApi | Testing upstream has been successfully completed
INFO, AdguardSvc, 38, 18.08.2021 20:39:40.978, Address https://dandelionsprout.asuscomm.com:2501/dns-query has been resolved with result True
INFO, AdguardSvc, 38, 18.08.2021 20:39:41.032, Adguard.Dns.Api.DnsApi | Testing upstream
INFO, AdguardSvc, 38, 18.08.2021 20:39:41.032, Adguard.Dns.Api.DnsApi | Start testing upstream Adguard.Dns.Api.DnsProxyServer.Configs.UpstreamOptions
INFO, AdguardSvc, 38, 18.08.2021 20:39:46.162, Adguard.Dns.Api.DnsApi | Testing upstream failed with an error Request failed (empty packet)
INFO, AdguardSvc, 38, 18.08.2021 20:39:46.162, Adguard.Dns.Api.DnsApi | Testing upstream has been successfully completed
INFO, AdguardSvc, 38, 18.08.2021 20:39:46.162, Address quic://dandelionsprout.asuscomm.com:48582 has been resolved with result False
INFO, AdguardSvc, 38, 18.08.2021 20:39:46.162, Testing DNS server has been completed successfully with the result False

Notably, the errors shown in those logs, differ from those that were in the predecessor issue reports.

ameshkov commented 3 years ago

@DandelionSprout just in case, are you on the latest nightly of AG for Android?

AGH has a newer QUIC version, higher than the current stable version of AG supports.

DandelionSprout commented 3 years ago

From what I can tell, yes: v4.0 nightly 22.

ameshkov commented 3 years ago

Hm, that's strange then.

Could you please enable debug log in AG for Android/Win and see if it starts printing additional info on the error?

DandelionSprout commented 3 years ago

AdGuard for Android debug log (Looks similar to the non-debug log in my eyes): adguard.log

AdGuard for Windows 7.7 nightly 13 verbose service-log (Also looks similar to me, sadly; also includes info on an unrelated but severe SSL certificate installation error): service_19-08-2021-13_27_23.367-2021-08-19.log

ameshkov commented 3 years ago

Tbh, I think the only viable way to troubleshoot this now is to use Wireshark on your Windows PC AND tcpdump on the AGH server and take a look at actual UDP packets on both sides.

DandelionSprout commented 3 years ago

I'm on the case.

! ———Wireshark (QUIC requests only, as I didn't know what else to look for.)———
! ——— 84.202.44.53 is my router's public IP. 192.168.1.168 is my DNS server's LAN IP. 192.168.1.188 is my PC with AdGuard for Windows. ———
11879   9.671864    192.168.1.188   84.202.44.53    QUIC    1274    Initial, DCID=8339928ebea2f44d2ae4584373307f4cf3f8, SCID=d07e374b6e110a5491beb304fcc93b400a, PKN: 0, CRYPTO, PADDING
11882   9.674080    84.202.44.53    192.168.1.188   QUIC    194 Retry, DCID=d07e374b6e110a5491beb304fcc93b400a, SCID=4206a1e7
11883   9.674190    192.168.1.188   84.202.44.53    QUIC    1274    Initial, DCID=4206a1e7, SCID=d07e374b6e110a5491beb304fcc93b400a, PKN: 1, CRYPTO, PADDING
11886   9.676154    192.168.1.168   192.168.1.188   QUIC    97  Initial, DCID=d07e374b6e110a5491beb304fcc93b400a, SCID=4206a1e7, PKN: 0, CC
12522   10.176299   192.168.1.188   84.202.44.53    QUIC    1274    Initial, DCID=4206a1e7, SCID=d07e374b6e110a5491beb304fcc93b400a, PKN: 2, CRYPTO, PADDING
12524   10.179608   192.168.1.168   192.168.1.188   QUIC    97  Initial, DCID=d07e374b6e110a5491beb304fcc93b400a, SCID=4206a1e7, PKN: 0, CC
14218   11.189265   192.168.1.188   84.202.44.53    QUIC    1274    Initial, DCID=4206a1e7, SCID=d07e374b6e110a5491beb304fcc93b400a, PKN: 3, CRYPTO, PADDING
14226   11.191422   192.168.1.168   192.168.1.188   QUIC    97  Initial, DCID=d07e374b6e110a5491beb304fcc93b400a, SCID=4206a1e7, PKN: 0, CC
16538   13.187478   192.168.1.188   84.202.44.53    QUIC    1274    Initial, DCID=4206a1e7, SCID=d07e374b6e110a5491beb304fcc93b400a, PKN: 4, CRYPTO, PADDING
16543   13.190047   192.168.1.168   192.168.1.188   QUIC    97  Initial, DCID=d07e374b6e110a5491beb304fcc93b400a, SCID=4206a1e7, PKN: 0, CC
23276   19.684265   192.168.1.188   84.202.44.53    QUIC    1274    Initial, DCID=5886a70450424098de9ac6f95b4be3ce09cd, SCID=e83b5fa6e7fdb039b1cb7a5dc3eff818d9, PKN: 0, CRYPTO, PADDING
23284   19.686454   84.202.44.53    192.168.1.188   QUIC    194 Retry, DCID=e83b5fa6e7fdb039b1cb7a5dc3eff818d9, SCID=a90b74db
23289   19.686583   192.168.1.188   84.202.44.53    QUIC    1274    Initial, DCID=a90b74db, SCID=e83b5fa6e7fdb039b1cb7a5dc3eff818d9, PKN: 1, CRYPTO, PADDING
23292   19.688472   192.168.1.168   192.168.1.188   QUIC    97  Initial, DCID=e83b5fa6e7fdb039b1cb7a5dc3eff818d9, SCID=a90b74db, PKN: 0, CC
23906   20.186259   192.168.1.188   84.202.44.53    QUIC    1274    Initial, DCID=a90b74db, SCID=e83b5fa6e7fdb039b1cb7a5dc3eff818d9, PKN: 2, CRYPTO, PADDING
23907   20.188317   192.168.1.168   192.168.1.188   QUIC    97  Initial, DCID=e83b5fa6e7fdb039b1cb7a5dc3eff818d9, SCID=a90b74db, PKN: 0, CC
25125   21.186081   192.168.1.188   84.202.44.53    QUIC    1274    Initial, DCID=a90b74db, SCID=e83b5fa6e7fdb039b1cb7a5dc3eff818d9, PKN: 3, CRYPTO, PADDING
25140   21.188347   192.168.1.168   192.168.1.188   QUIC    97  Initial, DCID=e83b5fa6e7fdb039b1cb7a5dc3eff818d9, SCID=a90b74db, PKN: 0, CC
28210   23.184121   192.168.1.188   84.202.44.53    QUIC    1274    Initial, DCID=a90b74db, SCID=e83b5fa6e7fdb039b1cb7a5dc3eff818d9, PKN: 4, CRYPTO, PADDING
28213   23.186139   192.168.1.168   192.168.1.188   QUIC    97  Initial, DCID=e83b5fa6e7fdb039b1cb7a5dc3eff818d9, SCID=a90b74db, PKN: 0, CC

! ——— tcpdump verbose ———
[dandelionsprout@fedora ~]$ sudo tcpdump port 48582 -v
[sudo] passord for dandelionsprout: 
dropped privs to tcpdump
tcpdump: listening on enp1s0u2c2, link-type EN10MB (Ethernet), snapshot length 262144 bytes
14:29:52.979807 IP (tos 0x0, ttl 54, id 17141, offset 0, flags [DF], proto TCP (6), length 583)
    thisis.feralhosting.com.48582 > fedora.36499: Flags [P.], cksum 0x0cc4 (correct), seq 3690508245:3690508788, ack 8803269, win 11975, length 543
14:29:52.979843 IP (tos 0x0, ttl 64, id 29399, offset 0, flags [DF], proto TCP (6), length 40)
    fedora.36499 > thisis.feralhosting.com.48582: Flags [.], cksum 0xccbd (correct), ack 543, win 24572, length 0
14:29:52.981066 IP (tos 0x0, ttl 64, id 29400, offset 0, flags [DF], proto TCP (6), length 583)
    fedora.36499 > thisis.feralhosting.com.48582: Flags [P.], cksum 0x5214 (correct), seq 1:544, ack 543, win 24576, length 543
14:29:53.021146 IP (tos 0x0, ttl 54, id 17142, offset 0, flags [DF], proto TCP (6), length 40)
    thisis.feralhosting.com.48582 > fedora.36499: Flags [.], cksum 0xfbd3 (correct), ack 544, win 11975, length 0
14:29:53.022594 IP (tos 0x0, ttl 127, id 33910, offset 0, flags [none], proto UDP (17), length 1260)
    _gateway.59243 > fedora.48582: UDP, length 1232
14:29:53.023335 IP (tos 0x0, ttl 64, id 56588, offset 0, flags [DF], proto UDP (17), length 180)
    fedora.48582 > _gateway.59243: UDP, length 152
14:29:53.026033 IP (tos 0x0, ttl 127, id 33911, offset 0, flags [none], proto UDP (17), length 1260)
    192.168.1.188.59243 > fedora.48582: UDP, length 1232
14:29:53.027096 IP (tos 0x0, ttl 64, id 1030, offset 0, flags [DF], proto UDP (17), length 83)
    fedora.48582 > 192.168.1.188.59243: UDP, length 55
14:29:53.156024 IP (tos 0x0, ttl 54, id 17143, offset 0, flags [DF], proto TCP (6), length 583)
    thisis.feralhosting.com.48582 > fedora.36499: Flags [P.], cksum 0xc441 (correct), seq 543:1086, ack 544, win 11975, length 543
14:29:53.156072 IP (tos 0x0, ttl 64, id 29401, offset 0, flags [DF], proto TCP (6), length 40)
    fedora.36499 > thisis.feralhosting.com.48582: Flags [.], cksum 0xc87f (correct), ack 1086, win 24572, length 0
14:29:53.525704 IP (tos 0x0, ttl 127, id 33912, offset 0, flags [none], proto UDP (17), length 1260)
    192.168.1.188.59243 > fedora.48582: UDP, length 1232
14:29:53.527272 IP (tos 0x0, ttl 64, id 1077, offset 0, flags [DF], proto UDP (17), length 83)
    fedora.48582 > 192.168.1.188.59243: UDP, length 55
14:29:54.525010 IP (tos 0x0, ttl 127, id 33913, offset 0, flags [none], proto UDP (17), length 1260)
    192.168.1.188.59243 > fedora.48582: UDP, length 1232
14:29:54.526183 IP (tos 0x0, ttl 64, id 1119, offset 0, flags [DF], proto UDP (17), length 83)
    fedora.48582 > 192.168.1.188.59243: UDP, length 55
14:29:56.523577 IP (tos 0x0, ttl 127, id 33919, offset 0, flags [none], proto UDP (17), length 1260)
    192.168.1.188.59243 > fedora.48582: UDP, length 1232
14:29:56.524755 IP (tos 0x0, ttl 64, id 1165, offset 0, flags [DF], proto UDP (17), length 83)
    fedora.48582 > 192.168.1.188.59243: UDP, length 55
14:29:58.365743 IP (tos 0x0, ttl 64, id 29402, offset 0, flags [DF], proto TCP (6), length 583)
    fedora.36499 > thisis.feralhosting.com.48582: Flags [P.], cksum 0x42a7 (correct), seq 544:1087, ack 1086, win 24576, length 543
14:29:58.407858 IP (tos 0x0, ttl 54, id 17144, offset 0, flags [DF], proto TCP (6), length 40)
    thisis.feralhosting.com.48582 > fedora.36499: Flags [.], cksum 0xf795 (correct), ack 1087, win 11975, length 0
14:29:59.773475 IP (tos 0x0, ttl 54, id 17145, offset 0, flags [DF], proto TCP (6), length 583)
    thisis.feralhosting.com.48582 > fedora.36499: Flags [P.], cksum 0x795f (correct), seq 1086:1629, ack 1087, win 11975, length 543
14:29:59.773535 IP (tos 0x0, ttl 64, id 29403, offset 0, flags [DF], proto TCP (6), length 40)
    fedora.36499 > thisis.feralhosting.com.48582: Flags [.], cksum 0xc441 (correct), ack 1629, win 24572, length 0
14:30:01.309965 IP (tos 0x0, ttl 127, id 33920, offset 0, flags [none], proto UDP (17), length 1260)
    _gateway.59252 > fedora.48582: UDP, length 1232
14:30:01.310800 IP (tos 0x0, ttl 64, id 56983, offset 0, flags [DF], proto UDP (17), length 180)
    fedora.48582 > _gateway.59252: UDP, length 152
14:30:01.313597 IP (tos 0x0, ttl 127, id 33921, offset 0, flags [none], proto UDP (17), length 1260)
    192.168.1.188.59252 > fedora.48582: UDP, length 1232
14:30:01.314587 IP (tos 0x0, ttl 64, id 1627, offset 0, flags [DF], proto UDP (17), length 83)
    fedora.48582 > 192.168.1.188.59252: UDP, length 55
14:30:01.813964 IP (tos 0x0, ttl 127, id 33922, offset 0, flags [none], proto UDP (17), length 1260)
    192.168.1.188.59252 > fedora.48582: UDP, length 1232
14:30:01.815119 IP (tos 0x0, ttl 64, id 1648, offset 0, flags [DF], proto UDP (17), length 83)
    fedora.48582 > 192.168.1.188.59252: UDP, length 55
14:30:02.827199 IP (tos 0x0, ttl 127, id 33923, offset 0, flags [none], proto UDP (17), length 1260)
    192.168.1.188.59252 > fedora.48582: UDP, length 1232
14:30:02.827878 IP (tos 0x0, ttl 64, id 1745, offset 0, flags [DF], proto UDP (17), length 83)
    fedora.48582 > 192.168.1.188.59252: UDP, length 55
14:30:04.829200 IP (tos 0x0, ttl 127, id 33924, offset 0, flags [none], proto UDP (17), length 1260)
    192.168.1.188.59252 > fedora.48582: UDP, length 1232
14:30:04.829777 IP (tos 0x0, ttl 64, id 1892, offset 0, flags [DF], proto UDP (17), length 83)
    fedora.48582 > 192.168.1.188.59252: UDP, length 55
14:30:26.593909 IP (tos 0x8, ttl 233, id 20612, offset 0, flags [DF], proto UDP (17), length 96)
    182.3.104.57.39222 > fedora.48582: UDP, length 68
14:30:26.619361 IP (tos 0x0, ttl 64, id 3549, offset 0, flags [DF], proto UDP (17), length 71)
    fedora.48582 > 182.3.104.57.39222: UDP, length 43
14:30:26.673074 IP (tos 0x0, ttl 64, id 3553, offset 0, flags [DF], proto UDP (17), length 248)
    fedora.48582 > 182.3.104.57.39222: UDP, length 220
14:30:26.913352 IP (tos 0x8, ttl 233, id 53310, offset 0, flags [DF], proto UDP (17), length 56)
    182.3.104.57.39222 > fedora.48582: UDP, length 28
14:30:26.913728 IP (tos 0x0, ttl 64, id 3561, offset 0, flags [DF], proto UDP (17), length 67)
    fedora.48582 > 182.3.104.57.39222: UDP, length 39
14:30:27.243192 IP (tos 0x8, ttl 233, id 22725, offset 0, flags [DF], proto UDP (17), length 57)
    182.3.104.57.39222 > fedora.48582: UDP, length 29
14:30:28.674641 IP (tos 0x8, ttl 233, id 31648, offset 0, flags [DF], proto UDP (17), length 92)
    182.3.104.57.39222 > fedora.48582: UDP, length 64
14:30:28.701332 IP (tos 0x0, ttl 64, id 3586, offset 0, flags [DF], proto UDP (17), length 71)
    fedora.48582 > 182.3.104.57.39222: UDP, length 43
14:30:28.708183 IP (tos 0x0, ttl 64, id 3587, offset 0, flags [DF], proto UDP (17), length 161)
    fedora.48582 > 182.3.104.57.39222: UDP, length 133
14:30:29.010228 IP (tos 0x8, ttl 233, id 63225, offset 0, flags [DF], proto UDP (17), length 56)
    182.3.104.57.39222 > fedora.48582: UDP, length 28
14:30:29.010918 IP (tos 0x0, ttl 64, id 3616, offset 0, flags [DF], proto UDP (17), length 67)
    fedora.48582 > 182.3.104.57.39222: UDP, length 39
14:30:29.314693 IP (tos 0x8, ttl 233, id 26775, offset 0, flags [DF], proto UDP (17), length 57)
    182.3.104.57.39222 > fedora.48582: UDP, length 29
14:30:43.554794 IP (tos 0x8, ttl 233, id 8685, offset 0, flags [DF], proto UDP (17), length 84)
    182.3.104.57.39222 > fedora.48582: UDP, length 56
14:30:43.581071 IP (tos 0x0, ttl 64, id 3882, offset 0, flags [DF], proto UDP (17), length 71)
    fedora.48582 > 182.3.104.57.39222: UDP, length 43
14:30:43.658933 IP (tos 0x0, ttl 64, id 3886, offset 0, flags [DF], proto UDP (17), length 204)
    fedora.48582 > 182.3.104.57.39222: UDP, length 176
14:30:43.954540 IP (tos 0x8, ttl 233, id 49439, offset 0, flags [DF], proto UDP (17), length 56)
    182.3.104.57.39222 > fedora.48582: UDP, length 28
14:30:43.955168 IP (tos 0x0, ttl 64, id 3915, offset 0, flags [DF], proto UDP (17), length 67)
    fedora.48582 > 182.3.104.57.39222: UDP, length 39
14:30:44.236843 IP (tos 0x8, ttl 233, id 15916, offset 0, flags [DF], proto UDP (17), length 57)
    182.3.104.57.39222 > fedora.48582: UDP, length 29
^C
48 packets captured
56 packets received by filter
0 packets dropped by kernel

Full Wireshark dump (Remove the .txt suffix before loading it; also note that it's 25 MB): Wireshark dump for AdGuard QUIC problems.pcapng.txt

ngorskikh commented 3 years ago

The issue is that your particular routing situation interferes with QUIC's address validation mechanism.

Probably, what's happening is that the first packet from your client to AGH server goes through NAT and arrives at AGH server as if it came from the router. After that, your router starts redirecting packets to AGH server transparently, so AGH server sees packets as if they are coming from the client directly, not the router.

This breaks QUIC's address validation mechanism:

AGH sees an Initial packet from _gateway (not 192.168.1.188, since it went through NAT).
AGH replies with a retry token which is specific for _gateway, and MUST be included in all subsequent Initial packets from that address.
Subsequent packets are routed differently, and now come from 192.168.1.188, so AGH rejects them since they carry a retry token for a different address.

It would be interesting to see a Wireshark capture from your AGH machine as well, to better understand what's going on.

DandelionSprout commented 3 years ago

Fedora claims to have an aarch64 version of Wireshark ready for download, so I'll see if I can try to install and run it sometime tonight.

ngorskikh commented 3 years ago

You could also try tcpdump -w <filename.pcap>

ghost commented 3 years ago

Yogadns Dnscrypt is simple.

DandelionSprout commented 3 years ago

Okay, I've now run Wireshark on my server. The output doesn't make all that much more sense to me (192.168.1.130 is my Android phone):

1965    3.319471919 192.168.1.1 192.168.1.168   QUIC    1274    Initial, DCID=46398300e3e5745f2e96bbb6a35fc805e0f4, SCID=b0140c0d42f045c0a0febc724226ea2ff0, PKN: 0, CRYPTO, PADDING
1966    3.320196650 192.168.1.168   192.168.1.1 QUIC    194 Retry, DCID=b0140c0d42f045c0a0febc724226ea2ff0, SCID=9cafddf2
1968    3.324200114 192.168.1.130   192.168.1.168   QUIC    1274    Initial, DCID=9cafddf2, SCID=b0140c0d42f045c0a0febc724226ea2ff0, PKN: 1, CRYPTO, PADDING
1969    3.325237156 192.168.1.168   192.168.1.130   QUIC    97  Initial, DCID=b0140c0d42f045c0a0febc724226ea2ff0, SCID=9cafddf2, PKN: 0, CC
2241    3.846250186 192.168.1.130   192.168.1.168   QUIC    1274    Initial, DCID=9cafddf2, SCID=b0140c0d42f045c0a0febc724226ea2ff0, PKN: 2, CRYPTO, PADDING
2242    3.846849252 192.168.1.168   192.168.1.130   QUIC    97  Initial, DCID=b0140c0d42f045c0a0febc724226ea2ff0, SCID=9cafddf2, PKN: 0, CC
2741    4.860325014 192.168.1.130   192.168.1.168   QUIC    1274    Initial, DCID=9cafddf2, SCID=b0140c0d42f045c0a0febc724226ea2ff0, PKN: 3, CRYPTO, PADDING
2742    4.861510775 192.168.1.168   192.168.1.130   QUIC    97  Initial, DCID=b0140c0d42f045c0a0febc724226ea2ff0, SCID=9cafddf2, PKN: 0, CC
3782    6.870938961 192.168.1.130   192.168.1.168   QUIC    1274    Initial, DCID=9cafddf2, SCID=b0140c0d42f045c0a0febc724226ea2ff0, PKN: 4, CRYPTO, PADDING
3783    6.871880930 192.168.1.168   192.168.1.130   QUIC    97  Initial, DCID=b0140c0d42f045c0a0febc724226ea2ff0, SCID=9cafddf2, PKN: 0, CC
10042   9.583268791 192.168.1.1 192.168.1.168   QUIC    1274    Initial, DCID=6966218c65af6a2e177ba8ba643a674bf1e1, SCID=b3bbc88b2b7340b0f9d9bd07fe469ea1df, PKN: 0, CRYPTO, PADDING
10045   9.584316296 192.168.1.168   192.168.1.1 QUIC    194 Retry, DCID=b3bbc88b2b7340b0f9d9bd07fe469ea1df, SCID=823bbc2e
10057   9.588243039 192.168.1.130   192.168.1.168   QUIC    1274    Initial, DCID=823bbc2e, SCID=b3bbc88b2b7340b0f9d9bd07fe469ea1df, PKN: 1, CRYPTO, PADDING
10060   9.589311969 192.168.1.168   192.168.1.130   QUIC    97  Initial, DCID=b3bbc88b2b7340b0f9d9bd07fe469ea1df, SCID=823bbc2e, PKN: 0, CC
11331   10.115823073    192.168.1.130   192.168.1.168   QUIC    1274    Initial, DCID=823bbc2e, SCID=b3bbc88b2b7340b0f9d9bd07fe469ea1df, PKN: 2, CRYPTO, PADDING
11335   10.117543402    192.168.1.168   192.168.1.130   QUIC    97  Initial, DCID=b3bbc88b2b7340b0f9d9bd07fe469ea1df, SCID=823bbc2e, PKN: 0, CC
13694   11.126729647    192.168.1.130   192.168.1.168   QUIC    1274    Initial, DCID=823bbc2e, SCID=b3bbc88b2b7340b0f9d9bd07fe469ea1df, PKN: 3, CRYPTO, PADDING
13698   11.128217830    192.168.1.168   192.168.1.130   QUIC    97  Initial, DCID=b3bbc88b2b7340b0f9d9bd07fe469ea1df, SCID=823bbc2e, PKN: 0, CC
18475   13.136758453    192.168.1.130   192.168.1.168   QUIC    1274    Initial, DCID=823bbc2e, SCID=b3bbc88b2b7340b0f9d9bd07fe469ea1df, PKN: 4, CRYPTO, PADDING
18480   13.137905197    192.168.1.168   192.168.1.130   QUIC    97  Initial, DCID=b3bbc88b2b7340b0f9d9bd07fe469ea1df, SCID=823bbc2e, PKN: 0, CC
24677   15.742278190    192.168.1.1 192.168.1.168   QUIC    1274    Initial, DCID=e60b4534ac6bccda91577d44a94ad8bc5c78, SCID=40e803d55f7b9e41ec2a8e56ca13b05d06, PKN: 0, CRYPTO, PADDING
24680   15.742948940    192.168.1.168   192.168.1.1 QUIC    194 Retry, DCID=40e803d55f7b9e41ec2a8e56ca13b05d06, SCID=9aaa8a81
24692   15.747355807    192.168.1.130   192.168.1.168   QUIC    1274    Initial, DCID=9aaa8a81, SCID=40e803d55f7b9e41ec2a8e56ca13b05d06, PKN: 1, CRYPTO, PADDING
24695   15.748004409    192.168.1.168   192.168.1.130   QUIC    97  Initial, DCID=40e803d55f7b9e41ec2a8e56ca13b05d06, SCID=9aaa8a81, PKN: 0, CC
25981   16.273349955    192.168.1.130   192.168.1.168   QUIC    1274    Initial, DCID=9aaa8a81, SCID=40e803d55f7b9e41ec2a8e56ca13b05d06, PKN: 2, CRYPTO, PADDING
25984   16.273987872    192.168.1.168   192.168.1.130   QUIC    97  Initial, DCID=40e803d55f7b9e41ec2a8e56ca13b05d06, SCID=9aaa8a81, PKN: 0, CC
28316   17.275650978    192.168.1.130   192.168.1.168   QUIC    1274    Initial, DCID=9aaa8a81, SCID=40e803d55f7b9e41ec2a8e56ca13b05d06, PKN: 3, CRYPTO, PADDING
28318   17.276256340    192.168.1.168   192.168.1.130   QUIC    97  Initial, DCID=40e803d55f7b9e41ec2a8e56ca13b05d06, SCID=9aaa8a81, PKN: 0, CC
29822   19.281977964    192.168.1.130   192.168.1.168   QUIC    1274    Initial, DCID=9aaa8a81, SCID=40e803d55f7b9e41ec2a8e56ca13b05d06, PKN: 4, CRYPTO, PADDING
29823   19.282910507    192.168.1.168   192.168.1.130   QUIC    97  Initial, DCID=40e803d55f7b9e41ec2a8e56ca13b05d06, SCID=9aaa8a81, PKN: 0, CC

Full log: Wireshark Raspberry Pi AGH server capture.zip (31 MB when it's unpacked)

I do however notice one aspect: There are no captured queries at all from my Windows PC (192.168.1.188).

ngorskikh commented 3 years ago

This seems to confirm the above hypothesis: the first initial packet comes from 192.168.1.1 as if the router did some sort of IP masquerading on output to the internal interface, while all subsequent packets come from 192.168.1.130, as if the router did DNAT on input from the internal interface.

To me this looks like some seriously broken router configuration :)

I've tried to reproduce this on my equipment, and my router correctly forwards the packets from the internal interface to the external one, then does masquerading, and then finally DNAT. For example this is what my AGH server sees when someone from the internal network tries accessing it via 123.123.123.123, which is the external (WAN) address of the router:

112 2021-08-22 18:54:02,557570  123.123.123.123 192.168.1.11    QUIC    1274    Initial, DCID=11d6990148d47319bacef3a2e507d53a9fa5, SCID=e6d11f0d9a46cef75c4a50c3b7df9624f2, PKN: 0, CRYPTO, PADDING
113 2021-08-22 18:54:02,557814  192.168.1.11    123.123.123.123 QUIC    100 Version Negotiation, DCID=e6d11f0d9a46cef75c4a50c3b7df9624f2, SCID=11d6990148d47319bacef3a2e507d53a9fa5
114 2021-08-22 18:54:02,563188  123.123.123.123 192.168.1.11    QUIC    1274    Initial, DCID=17cef632a21bc5eb32f47265bddab83701b4, SCID=fd9f0680d5eee528e78009819c877dd1cc, PKN: 0, CRYPTO, PADDING
115 2021-08-22 18:54:02,563469  192.168.1.11    123.123.123.123 QUIC    194 Retry, DCID=fd9f0680d5eee528e78009819c877dd1cc, SCID=9d300575
116 2021-08-22 18:54:02,567608  123.123.123.123 192.168.1.11    QUIC    1274    Initial, DCID=9d300575, SCID=fd9f0680d5eee528e78009819c877dd1cc, PKN: 1, CRYPTO, PADDING
117 2021-08-22 18:54:02,575669  192.168.1.11    123.123.123.123 QUIC    1294    Handshake, DCID=fd9f0680d5eee528e78009819c877dd1cc, SCID=eb762aeb
118 2021-08-22 18:54:02,575781  192.168.1.11    123.123.123.123 QUIC    1187    Handshake, DCID=fd9f0680d5eee528e78009819c877dd1cc, SCID=eb762aeb
119 2021-08-22 18:54:02,575803  192.168.1.11    123.123.123.123 QUIC    198 Protected Payload (KP0), DCID=fd9f0680d5eee528e78009819c877dd1cc

Here are some of the possible solutions:

Try to fix the router configuration (depending on your level of geekitude, you could try SSHing into the router and seeing what the hell they did to the iptables; or just poking around in their web interface, especially around things that might be named something like "Port forwarding", "NAT" or "DMZ")
Make it so that the domain name of your AGH server resolves to an internal address from the internal network. You do have some sort of local plain DNS server as a bootstrap, right? It's probably even the AGH itself, in which case you could just write a rule to resolve dandelionsprout.asuscomm.com to 192.168.1.168 for internal clients to avoid all this routing strangeness.

DandelionSprout commented 3 years ago

My router is an ASUS RT-AC68U. It's becoming pretty old (2013 model), although it has almost all of the functions of more recent ASUS routers. I've telnet-ed into it on one previous occasion, but it's difficult to maneuver in.

As for the two suggestions, it should be decently possible for me to look into those various re-routing methods within 48 hours, possibly also including the Linux server's /etc/hosts file.

DandelionSprout commented 3 years ago

After 1½ hours, I've found out the underlying reason for this problem, which in turn lacks a solution:

The AdGuard Home alpha version (v0.107.0-a.136+550b1798) is lying about listening to port 48582 on LAN:

This can be confirmed when I telnet that port, which gives a connection error.

ameshkov commented 3 years ago

@DandelionSprout QUIC works over UDP and telnet tries to connect using TCP.

netstat command on your screenshot does not print those that listen to UDP at all due to grep LISTEN probably.

DandelionSprout commented 3 years ago

Ah, okay. Now I see it.

So now I'm in a bit of a problem. I genuinely can't figure out what the problem is, after trying some 60 different things, and I've had a fever for 4 days now on top of that.

ngorskikh commented 3 years ago

What about just resolving your domain to a private address from the private network? That would solve your routing issues. To be clear, here's how that would work:

Configure AGH to listen for plain DNS (maybe only on private network)
Configure DHCP on your router to advertise AGH as the DNS server
Add a client tag in AGH for your private network (e.g. 192.168.0.0/16 or 192.168.1.0/24)
Add a rule like your.domain.com$client=YourClientNetTag,dnsrewrite=192.168.1.168 This should work assuming the clients use the DHCP DNS settings to bootstrap the QUIC resolver (which they should do, but in AdGuard apps the bootstrap is also configurable).

DandelionSprout commented 3 years ago

Trying steps 3 and 4 while skipping steps 1 and 2, through dandelionsprout.asuscomm.com^$dnsrewrite=192.168.1.168,client='Imres Moto One Zoom' seemed to work, but with the fairly major drawback that it causes DoH/DoT connection errors to my secondary server at 192.168.1.188.

I guess I'll have to try to follow steps 1 and 2 as well.

Update 19:37 UTC: Turns out that my ASUS router is unable to use a LAN AGH server as both of its IPv4 DNS servers, as it'll for some reason cut the internet connection.

Additionally, trying to use two $dnsrewrite entries doesn't work, since AdGuard for Android insist on getting a working connection from both IPs for all connections.

dandelionsprout.asuscomm.com$dnsrewrite=192.168.1.168,client='Imres Moto One Zoom',dnstype=A
dandelionsprout.asuscomm.com$dnsrewrite=192.168.1.188,client='Imres Moto One Zoom',dnstype=A

DandelionSprout commented 3 years ago

I was at least finally able to find the router's netstat table, as was requested:

ngorskikh commented 3 years ago

Regarding

Update 19:37 UTC: Turns out that my ASUS router is unable to use a LAN AGH server as both of its IPv4 DNS servers, as it'll for some reason cut the internet connection.

, if your AGH server is configured to use a non-plain upstream, and it gets it's DNS settings from DHCP, too, this would lead to an infinite loop as AGH would try to bootstrap it's upstream using itself :) Just configure your AGH to use any public DNS of your choosing instead of those from DHCP.

ameshkov commented 3 years ago

Closing this since this is not a bug

AdguardTeam / AdGuardHome