polarathene
commented
4 weeks ago Resolved
I now consider this resolved given the findings I've documented here below.
TL:DR (Update):
- Besides the additional observations noted in the summaries below, without setting the nameserver to use
1.1.1.1the queries were all being sent to router DNS (192.168.1.1below) indirectly for WSL2 and Docker. - The router DNS refuses queries over TCP, but
TXT github.comis too big for standard UDP query (512 byte size limit), while1.1.1.1does support TCP, and a truncated response informs the DNS client to request again over TCP where it succeeds.
I am not sure how to reproduce the original error I reported in the issue title, but it does appear like it was specific to my environment and router / DNS concerns discovered that I've noted in the summary.
Summary:
- WSL2 and Docker have separate DNS services that routed to the one the Windows 11 host was using AFAIK.
- WSL2 DNS appears to have some cache layer present while the other two did not? (or at least the WSL2 one was distinct / prioritized)
- Windows 11 when using the router issued DNS server, would emit an error for the
TXT github.comquery. The WSL2 / Docker DNS services did not bubble that up for visibility. - Windows 11 had
0.0.0.0configured as a 2ndry DNS service (even if there was no alternative DNS service available), where DNS queries could be observed being sent across both. When the 2nd DNS service was1.1.1.1both Windows and WSL2 output correct results forTXT github.com, but the Docker container did not. Presumably the difference here is how they integrate/relay with the host DNS that they forward the queries to and receive responses from.
Summary with a bit more context/insights:
- Despite the layers (Host / WSL2 / Docker), with the latter two having DNS servers that behave differently and the host being direct to the router DNS server supplied via DHCP, they all were connecting through the same DNS service on the router.
-
At least with the attempted reproduction below, it was discovered that the Windows host had the router DNS as the primary DNS with a secondary DNS as
0.0.0.0. Two insights were found related to this:- The two services were both queried (at least visible with
doggo --debugon the Windows 11 host output, WSL2 and Docker were unaware), and the query appeared to be in parallel which explains why my original investigation into DNS issues were racey with inconsistent results. 0.0.0.0would resolve to127.0.0.1for DNS service. I had CoreDNS (started from WSL2 via Docker container) that had a fallback rule to forward unresolved queries to1.1.1.1. It's possible this container was also running in the background when I originally reported, but I was assuming without anything explicitly configured to use it at both times that it was not receiving queries.-
The published port
53:53for the container was not accessible within WSL2 network interfaces.- Docker seems to have avoided that due to loopback listening at
127.0.0.53:53? (Docker runs it's own DNS somewhere at127.0.0.11:53too IIRC, perhaps on the WSL2 docker image that Docker Desktop manages). - WSL2 interfaces were all potentially ignored due to that conflict for the implicit default bind address
0.0.0.0(all addresses) with the docker port publishing syntax. If I instead explicitly configure as127.0.0.1:53:53thenss -tulpnlists the binding within WSL2. Doing so makes the query fail on Windows 11 host (PowerShell), whilst I can bind to the private network IP assigned to that WSL2 distro instead, and explicitly query that for Windows 11 and Docker containers to reach. 127.0.0.53in WSL2 seems to forward to/etc/resolv.confwhen queried, thus behaves the same as when querying without an explicit name server.
Error output for
TXT github.comquery has fluctuated between these two (probably due parallel query on host?):$ ./doggo TXT github.com @127.0.0.53 time=2024-08-07T14:36:03.986+12:00 level=ERROR msg="error in lookup" error="read udp 127.0.0.1:42247->127.0.0.53:53: i/o timeout"$ ./doggo TXT github.com time=2024-08-07T14:38:59.708+12:00 level=ERROR msg="Error looking up DNS records" error="context deadline exceeded"A github.comquery works fine. Pretty sure theTXTrecord failure is related to the large record that is technically multi-part and the router DNS server just doesn't handle that (it's a budget router known to have issues that require manual restarts). - Docker seems to have avoided that due to loopback listening at
- The two services were both queried (at least visible with
- When isolating the query specifically to the router issued DNS it reliably produced the failures for
TXT github.comqueries. WSL2 and the Docker container could all reach192.168.1.1to produce this error that only Windows 11 was originally outputting. Presumably from the indirection that error did not bubble back up through the WSL2 and Docker internal DNS services.
For context my WSL2 install is not the newer 2.2.1 release (which has DNS improvements that'd likely change the findings reported here):
$ wsl --version
WSL version: 2.0.0.0
Kernel version: 5.15.123.1-1
WSLg version: 1.0.57
MSRDC version: 1.2.4485
Direct3D version: 1.608.2-61064218
DXCore version: 10.0.25880.1000-230602-1350.main
Windows version: 10.0.22631.3810If you landed here due to dns: overflowing header size, it seems this topic is discussed well at CoreDNS, the query like TXT github.com over UDP is larger than 512 bytes (for github query is was over 1600), but that should implicitly query again over TCP as a workaround if the response had a truncate bit (tc):
- https://github.com/coredns/coredns/issues/5953
- https://github.com/coredns/coredns/pull/6277
- https://github.com/coredns/coredns/issues/5998
- https://github.com/coredns/coredns/issues/6149#issue-1747028110
https://github.com/coredns/coredns/pull/6003#discussion_r1154812126
// If the error is an overflow, we probably have a misbehaving upstream that is sending out-of-spec (>512 byte) UDP responses without an eDNS0 OPT RR. // Set the truncate bit and return the result
dig/doggo `TXT github.com` queries
The DNS service from the router in this comment (`192.168.1.1`) below completely fails at the `TXT github.com` query: ```console # NOTE: The failure output was actually repeated 3 times: $ dig TXT github.com @192.168.1.1 ;; Truncated, retrying in TCP mode. ;; Connection to 192.168.1.1#53(192.168.1.1) for github.com failed: connection refused. ;; no servers could be reached ``` It would appear the router DNS **does not support TCP queries**, which was required for that query. Smaller query responses that are within the 512 byte buffer size complete fine over UDP: ```console $ dig A github.com @192.168.1.1 ;; Warning: Message parser reports malformed message packet. ; <<>> DiG 9.18.28-0ubuntu0.22.04.1-Ubuntu <<>> A github.com @192.168.1.1 ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 43272 ;; flags: qr; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1 ;; QUESTION SECTION: ;github.com. IN A ;; ANSWER SECTION: . 0 CLASS1232 OPT 10 8 BdgO5ddpmWE= ;; ADDITIONAL SECTION: github.com. 48 IN A 4.237.22.38 ;; Query time: 10 msec ;; SERVER: 192.168.1.1#53(192.168.1.1) (UDP) ;; WHEN: Wed Aug 07 19:43:26 NZST 2024 ;; MSG SIZE rcvd: 67 ``` ```console $ ./doggo A github.com @udp://192.168.1.1 NAME TYPE CLASS TTL ADDRESS NAMESERVER github.com. A IN 8s 4.237.22.38 192.168.1.1:53 # Router doesn't support TCP queries: $ ./doggo A github.com @tcp://192.168.1.1 time=2024-08-07T19:46:03.982+12:00 level=ERROR msg="error in lookup" error="dial tcp 192.168.1.1:53: connect: connection refused" NAME TYPE CLASS TTL ADDRESS NAMESERVER ``` WSL2 (`172.18.160.1`) has the equivalent response of querying `192.168.1.1` directly (_same with the `dig TXT github.com` query too_): ```console $ ./doggo A github.com @udp://172.18.160.1 NAME TYPE CLASS TTL ADDRESS NAMESERVER github.com. A IN 0s 4.237.22.38 172.18.160.1:53 $ ./doggo A github.com @tcp://172.18.160.1 time=2024-08-07T07:53:08.521Z level=ERROR msg="error in lookup" error="dial tcp 172.18.160.1:53: connect: connection refused" NAME TYPE CLASS TTL ADDRESS NAMESERVER ``` Docker / DockerDesktop (`192.168.65.7`) doesn't, TCP was accepted and resolved, but both TCP and UDP fail silently when querying `TXT github.com`. ```console $ ./doggo A github.com @udp://192.168.65.7 NAME TYPE CLASS TTL ADDRESS NAMESERVER github.com. A IN 22s 4.237.22.38 192.168.65.7:53 $ ./doggo A github.com @tcp://192.168.65.7 NAME TYPE CLASS TTL ADDRESS NAMESERVER github.com. A IN 32s 4.237.22.38 192.168.65.7:53 ``` ```console dig TXT github.com @192.168.65.7 ; <<>> DiG 9.18.24 <<>> TXT github.com @192.168.65.7 ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 51445 ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;github.com. IN TXT ;; Query time: 1920 msec ;; SERVER: 192.168.65.7#53(192.168.65.7) (UDP) ;; WHEN: Wed Aug 07 08:02:45 UTC 2024 ;; MSG SIZE rcvd: 28 ```Original Response (full details / process)
As an update to this, I tried again to reproduce out of curiosity. I found that the latest release of doggo did not output the error, just blank results.
I wondered if this was a change related to doggo so I double-checked with q, which still has not published any newer releases, so for reference q in both cases was definitely 0.19.2 (Jan 2024), and I used the equivalent GHCR image. This too produced empty output.
$ ./doggo TXT github.com
NAME TYPE CLASS TTL ADDRESS NAMESERVER
# Doesn't look like doggo is aware of the issue:
$ ./doggo --debug TXT github.com
time=2024-08-06T22:49:44.342Z level=DEBUG msg="LoadNameservers: Initial nameservers" nameservers=[]
time=2024-08-06T22:49:44.342Z level=DEBUG msg="No user specified nameservers, falling back to system nameservers"
time=2024-08-06T22:49:44.342Z level=DEBUG msg="Loaded system nameservers" nameservers="[{Address:192.168.65.7:53 Type:udp}]"
time=2024-08-06T22:49:44.342Z level=DEBUG msg="initiating UDP resolver"
time=2024-08-06T22:49:44.342Z level=DEBUG msg="Attempting to resolve" domain=github.com. ndots=0 nameserver=192.168.65.7:53
# Neither output anything:
$ docker run --rm -it ghcr.io/natesales/q TXT github.com --txtconcat
$ docker run --rm -it ghcr.io/natesales/q TXT github.comSo given that q hasn't changed, perhaps something else has.
- My system has been rebooted since the original report, but I did this test with the Windows host assigning DNS server via automatic DHCP (thus the router / ISP issued DNS). I don't think WSL2 was updated.
- I may have updated Docker, although the current version installed is
26.1.1(April 2024 release).
With doggo 1.0.4:
Windows 11 host (PowerShell)
```console $ ./doggo.exe TXT github.com time=2024-08-07T11:08:41.017+12:00 level=ERROR msg="error in lookup" error="dial tcp 192.168.1.1:53: connectex: No connection could be made because the target machine actively refused it." NAME TYPE CLASS TTL ADDRESS NAMESERVER github.com. TXT IN 2087s "MS=6BF03E6AF5CB689E315FB6199603BABF2C88D805" 0.0.0.0:53 github.com. TXT IN 2087s "MS=ms44452932" 0.0.0.0:53 github.com. TXT IN 2087s "MS=ms58704441" 0.0.0.0:53 github.com. TXT IN 2087s "adobe-idp-site-verification=b92c9e999aef825edc36e0a3d847d2dbad5b2fc0e05c79ddd7a16139b48ecf4b" 0.0.0.0:53 github.com. TXT IN 2087s "apple-domain-verification=RyQhdzTl6Z6x8ZP4" 0.0.0.0:53 github.com. TXT IN 2087s "atlassian-domain-verification=jjgw98AKv2aeoYFxiL/VFaoyPkn3undEssTRuMg6C/3Fp/iqhkV4HVV7WjYlVeF8" 0.0.0.0:53 github.com. TXT IN 2087s "beautifulai-site-verification=e478d764-9335-4af3-ac7a-2d5ab61b59aa" 0.0.0.0:53 github.com. TXT IN 2087s "calendly-site-verification=at0DQARi7IZvJtXQAWhMqpmIzpvoBNF7aam5VKKxP" 0.0.0.0:53 github.com. TXT IN 2087s "docusign=087098e3-3d46-47b7-9b4e-8a23028154cd" 0.0.0.0:53 github.com. TXT IN 2087s "facebook-domain-verification=39xu4jzl7roi7x0n93ldkxjiaarx50" 0.0.0.0:53 github.com.\BrenTXT DownIN \do2087s.0."google-site-verification=82Le34Flgtd15ojYhHlGF_6g72muSjamlMVThBOJpks" 0.0.0.0:53 github.com. TXT IN 2087s "google-site-verification=UTM-3akMgubp6tQtgEuAkYNYLyYAvpTnnSrDMWoDR3o" 0.0.0.0:53 github.com. TXT IN 2087s "krisp-domain-verification=ZlyiK7XLhnaoUQb2hpak1PLY7dFkl1WE" 0.0.0.0:53 github.com. TXT IN 2087s "loom-site-verification=f3787154f1154b7880e720a511ea664d" 0.0.0.0:53 github.com. TXT IN 2087s "miro-verification=d2e174fdb00c71e0bcf58f8e58c3da2dd80dcfa9" 0.0.0.0:53 github.com. TXT IN 2087s "stripe-verification=f88ef17321660a01bab1660454192e014defa29ba7b8de9633c69d6b4912217f" 0.0.0.0:53 github.com. TXT IN 2087s "v=spf1 ip4:192.30.252.0/22 0.0.0.0:53 include:_netblocks.google.com include:_netblocks2.google.com include:_netblocks3.google.com include:spf.protection.outlook.com include:mail.zendesk.com include:_spf.salesforce.com include:servers.mcsv.net ip4:166.78.69.169 ip4:1" "66.78.69.170 ip4:166.78.71.131 ip4:167.89.101.2 ip4:167.89.101.192/28 ip4:192.254.112.60 ip4:192.254.112.98/31 ip4:192.254.113.10 ip4:192.254.113.101 ip4:192.254.114.176 ip4:62.253.227.114 ~all" ```WSL2 Ubuntu 22.04.2 (Kernel: 5.15.123.1-microsoft-standard-WSL2)
For context this is the default distro WSL2 installed, while Docker Desktop installs Docker into a separate distro, whilst you interact with it from other WSL2 distro installs like Ubuntu here (_thus a bit more complexity vs a plain linux host with Docker_). ```console $ ./doggo A AAAA NS MX TXT CNAME github.com NAME TYPE CLASS TTL ADDRESS NAMESERVER github.com. A IN 0s 4.237.22.38 172.18.160.1:53 github.com. SOA IN 3578s dns1.p08.nsone.net. 172.18.160.1:53 hostmaster.nsone.net. 1656468023 43200 7200 1209600 3600 github.com. NS IN 0s dns1.p08.nsone.net. 172.18.160.1:53 github.com. NS IN 0s dns2.p08.nsone.net. 172.18.160.1:53 github.com. NS IN 0s dns3.p08.nsone.net. 172.18.160.1:53 github.com. NS IN 0s dns4.p08.nsone.net. 172.18.160.1:53 github.com. NS IN 0s ns-1283.awsdns-32.org. 172.18.160.1:53 github.com. NS IN 0s ns-1707.awsdns-21.co.uk. 172.18.160.1:53 github.com. NS IN 0s ns-421.awsdns-52.com. 172.18.160.1:53 github.com. NS IN 0s ns-520.awsdns-01.net. 172.18.160.1:53 github.com. MX IN 0s 1 aspmx.l.google.com. 172.18.160.1:53 github.com. MX IN 0s 5 alt1.aspmx.l.google.com. 172.18.160.1:53 github.com. MX IN 0s 5 alt2.aspmx.l.google.com. 172.18.160.1:53 github.com. MX IN 0s 10 alt3.aspmx.l.google.com. 172.18.160.1:53 github.com. MX IN 0s 10 alt4.aspmx.l.google.com. 172.18.160.1:53 github.com. TXT IN 3376s "MS=6BF03E6AF5CB689E315FB6199603BABF2C88D805" 172.18.160.1:53 github.com. TXT IN 3376s "MS=ms44452932" 172.18.160.1:53 github.com. TXT IN 3376s "MS=ms58704441" 172.18.160.1:53 github.com. TXT IN 3376s "adobe-idp-site-verification=b92c9e999aef825edc36e0a3d847d2dbad5b2fc0e05c79ddd7a16139b48ecf4b" 172.18.160.1:53 github.com. TXT IN 3376s "apple-domain-verification=RyQhdzTl6Z6x8ZP4" 172.18.160.1:53 github.com. TXT IN 3376s "atlassian-domain-verification=jjgw98AKv2aeoYFxiL/VFaoyPkn3undEssTRuMg6C/3Fp/iqhkV4HVV7WjYlVeF8" 172.18.160.1:53 github.com. TXT IN 3376s "beautifulai-site-verification=e478d764-9335-4af3-ac7a-2d5ab61b59aa" 172.18.160.1:53 github.com. TXT IN 3376s "calendly-site-verification=at0DQARi7IZvJtXQAWhMqpmIzpvoBNF7aam5VKKxP" 172.18.160.1:53 github.com. TXT IN 3376s "docusign=087098e3-3d46-47b7-9b4e-8a23028154cd" 172.18.160.1:53 github.com. TXT IN 3376s "facebook-domain-verification=39xu4jzl7roi7x0n93ldkxjiaarx50" 172.18.160.1:53 github.com. TXT IN 3376s "google-site-verification=82Le34Flgtd15ojYhHlGF_6g72muSjamlMVThBOJpks" 172.18.160.1:53 github.com. TXT IN 3376s "google-site-verification=UTM-3akMgubp6tQtgEuAkYNYLyYAvpTnnSrDMWoDR3o" 172.18.160.1:53 github.com. TXT IN 3376s "krisp-domain-verification=ZlyiK7XLhnaoUQb2hpak1PLY7dFkl1WE" 172.18.160.1:53 github.com. TXT IN 3376s "loom-site-verification=f3787154f1154b7880e720a511ea664d" 172.18.160.1:53 github.com. TXT IN 3376s "miro-verification=d2e174fdb00c71e0bcf58f8e58c3da2dd80dcfa9" 172.18.160.1:53 github.com. TXT IN 3376s "stripe-verification=f88ef17321660a01bab1660454192e014defa29ba7b8de9633c69d6b4912217f" 172.18.160.1:53 github.com. TXT IN 3376s "v=spf1 ip4:192.30.252.0/22 172.18.160.1:53 include:_netblocks.google.com include:_netblocks2.google.com include:_netblocks3.google.com include:spf.protection.outlook.com include:mail.zendesk.com include:_spf.salesforce.com include:servers.mcsv.net ip4:166.78.69.169 ip4:1" "66.78.69.170 ip4:166.78.71.131 ip4:167.89.101.2 ip4:167.89.101.192/28 ip4:192.254.112.60 ip4:192.254.112.98/31 ip4:192.254.113.10 ip4:192.254.113.101 ip4:192.254.114.176 ip4:62.253.227.114 ~all" github.com. SOA IN 900s ns-1707.awsdns-21.co.uk. 172.18.160.1:53 awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400 ```So both Windows 11 host and WSL2 have no issue with the TXT github.com query. The windows 11 host does emit a lookup error before outputting results however.
192.168.1.1 for that error is assigned as the DNS server to my laptop wifi interface that is handling the internet connection. So querying it explicitly:
# Need to quote for PowerShell, else syntax error due to shell feature:
$ ./doggo.exe TXT github.com '@192.168.1.1'
time=2024-08-07T11:20:42.897+12:00 level=ERROR msg="error in lookup" error="dial tcp 192.168.1.1:53: connectex: No connection could be made because the target machine actively refused it."
NAME TYPE CLASS TTL ADDRESS NAMESERVERLikewise when done on WSL2:
$ ./doggo TXT github.com @192.168.1.1
time=2024-08-07T11:22:00.427+12:00 level=ERROR msg="error in lookup" error="dial tcp 192.168.1.1:53: connect: connection refused"
NAME TYPE CLASS TTL ADDRESS NAMESERVERAnd for additional context, within a Docker container (volume mounting the same doggo 1.0.4 github release binary used with WSL2):
$ docker run --rm -v ./doggo:/opt/doggo:ro --workdir /opt alpine ash -c './doggo TXT github.com @192.168.1.1'
time=2024-08-06T23:23:01.328Z level=ERROR msg="error in lookup" error="dial tcp 192.168.1.1:53: connect: connection refused"
NAME TYPE CLASS TTL ADDRESS NAMESERVERAnd just so it's perfectly clear with that obvious cause, this only affects the TXT record query, the docker container (or rather 192.168.1.1 DNS server) supports querying other records without issue (no error emitted):
$ docker run --rm -v ./doggo:/opt/doggo:ro --workdir /opt alpine ash -c './doggo A github.com @192.168.1.1'
NAME TYPE CLASS TTL ADDRESS NAMESERVER
github.com. A IN 3s 4.237.22.38 192.168.1.1:53So some difference between A and TXT query there under the hood seems relevant?
On the Windows 11 when checking for the DNS Server it was configured with via PowerShell, there is this snippet:
$ ipconfig /all
# I've omitted/trimmed fields, but note the 2nd entry for DNS servers:
Wireless LAN adapter Wi-Fi:
Default Gateway . . . . . . . . . : 192.168.1.1
DHCP Server . . . . . . . . . . . : 192.168.1.1
DNS Servers . . . . . . . . . . . : 192.168.1.1
0.0.0.0Running doggo via PowerShell with debug this time, we get this information before the results are printed:
$ ./doggo.exe --debug TXT github.com
time=2024-08-07T11:28:32.961+12:00 level=DEBUG msg="LoadNameservers: Initial nameservers" nameservers=[]
time=2024-08-07T11:28:32.961+12:00 level=DEBUG msg="No user specified nameservers, falling back to system nameservers"
time=2024-08-07T11:28:32.967+12:00 level=DEBUG msg="Loaded system nameservers" nameservers="[{Address:192.168.1.1:53 Type:udp} {Address:0.0.0.0:53 Type:udp}]"
time=2024-08-07T11:28:32.968+12:00 level=DEBUG msg="initiating UDP resolver"
time=2024-08-07T11:28:32.968+12:00 level=DEBUG msg="initiating UDP resolver"
time=2024-08-07T11:28:32.968+12:00 level=DEBUG msg="Attempting to resolve" domain=github.com. ndots=0 nameserver=0.0.0.0:53
time=2024-08-07T11:28:32.968+12:00 level=DEBUG msg="Attempting to resolve" domain=github.com. ndots=0 nameserver=192.168.1.1:53
time=2024-08-07T11:28:32.977+12:00 level=DEBUG msg="Response truncated; retrying now" protocol=tcp
time=2024-08-07T11:28:32.977+12:00 level=DEBUG msg="Attempting to resolve" domain=github.com. ndots=0 nameserver=192.168.1.1:53
time=2024-08-07T11:28:33.009+12:00 level=DEBUG msg="Response truncated; retrying now" protocol=tcp
time=2024-08-07T11:28:33.009+12:00 level=DEBUG msg="Attempting to resolve" domain=github.com. ndots=0 nameserver=0.0.0.0:53
time=2024-08-07T11:28:35.036+12:00 level=ERROR msg="error in lookup" error="dial tcp 192.168.1.1:53: connectex: No connection could be made because the target machine actively refused it."0.0.0.0 is valid for bind/listen address to cover all interfaces, but typically as a target for requests IIRC? The ipconfig output showed no other configured DNS services either so I'm a bit curious what happened differently here. Seems from the above debug logs that both queries were sent in parallel (which explains why in May when I raised this bug report I had weird/inconsistent/racey results when resolving crates.io records with q unless I explicit set the DNS server to something like 1.1.1.1)..
# No error this time:
$ ./doggo.exe TXT github.com '@0.0.0.0'
NAME TYPE CLASS TTL ADDRESS NAMESERVER
github.com. TXT IN 578s "MS=6BF03E6AF5CB689E315FB6199603BABF2C88D805" 0.0.0.0:53
github.com. TXT IN 578s "MS=ms44452932" 0.0.0.0:53
github.com. TXT IN 578s "MS=ms58704441" 0.0.0.0:53
github.com. TXT IN 578s "adobe-idp-site-verification=b92c9e999aef825edc36e0a3d847d2dbad5b2fc0e05c79ddd7a16139b48ecf4b" 0.0.0.0:53
github.com. TXT IN 578s "apple-domain-verification=RyQhdzTl6Z6x8ZP4" 0.0.0.0:53
github.com. TXT IN 578s "atlassian-domain-verification=jjgw98AKv2aeoYFxiL/VFaoyPkn3undEssTRuMg6C/3Fp/iqhkV4HVV7WjYlVeF8" 0.0.0.0:53
github.com. TXT IN 578s "beautifulai-site-verification=e478d764-9335-4af3-ac7a-2d5ab61b59aa" 0.0.0.0:53
github.com. TXT IN 578s "calendly-site-verification=at0DQARi7IZvJtXQAWhMqpmIzpvoBNF7aam5VKKxP" 0.0.0.0:53
github.com. TXT IN 578s "docusign=087098e3-3d46-47b7-9b4e-8a23028154cd" 0.0.0.0:53
github.com. TXT IN 578s "facebook-domain-verification=39xu4jzl7roi7x0n93ldkxjiaarx50" 0.0.0.0:53
github.com. TXT IN 578s "google-site-verification=82Le34Flgtd15ojYhHlGF_6g72muSjamlMVThBOJpks" 0.0.0.0:53
github.com. TXT IN 578s "google-site-verification=UTM-3akMgubp6tQtgEuAkYNYLyYAvpTnnSrDMWoDR3o" 0.0.0.0:53
github.com. TXT IN 578s "krisp-domain-verification=ZlyiK7XLhnaoUQb2hpak1PLY7dFkl1WE" 0.0.0.0:53
github.com. TXT IN 578s "loom-site-verification=f3787154f1154b7880e720a511ea664d" 0.0.0.0:53
github.com. TXT IN 578s "miro-verification=d2e174fdb00c71e0bcf58f8e58c3da2dd80dcfa9" 0.0.0.0:53
github.com. TXT IN 578s "stripe-verification=f88ef17321660a01bab1660454192e014defa29ba7b8de9633c69d6b4912217f" 0.0.0.0:53
github.com. TXT IN 578s "v=spf1 ip4:192.30.252.0/22 0.0.0.0:53
include:_netblocks.google.com
include:_netblocks2.google.com
include:_netblocks3.google.com
include:spf.protection.outlook.com
include:mail.zendesk.com
include:_spf.salesforce.com
include:servers.mcsv.net
ip4:166.78.69.169 ip4:1"
"66.78.69.170 ip4:166.78.71.131
ip4:167.89.101.2
ip4:167.89.101.192/28
ip4:192.254.112.60
ip4:192.254.112.98/31
ip4:192.254.113.10
ip4:192.254.113.101
ip4:192.254.114.176
ip4:62.253.227.114 ~all"Not the easiest thing to search for, but thankfully trying the same in linux land we get some insight:
$ docker run --rm -v ./doggo:/opt/doggo:ro --workdir /opt alpine ash -c './doggo A github.com @0.0.0.0'
time=2024-08-06T23:43:54.217Z level=ERROR msg="error in lookup" error="read udp 127.0.0.1:54335->127.0.0.1:53: read: connection refused"
NAME TYPE CLASS TTL ADDRESS NAMESERVERAnd sure enough, on Windows 11 with powershell if I adjust that to query @127.0.0.1 the query succeeds. For additional context, on WSL2 I get the same error as shown within the Docker command output.
Now I can't recall if the following would apply to my original report in May (although it's now clear how the DNS server IP was likely the same as the one on the host as shown, but I had changed the host to use 1.1.1.1, the Docker Daemon had not been restarted and thus might have tried to still query DNS from the router IP, or the original error may have been related to Docker embedded DNS and potentially it's own fallback, which may have been fixed if I have since upgraded Docker 🤷♂️ )
This additional DNS service at 127.0.0.1 on the host that neither WSL2 or the Docker container could reach (since duh, 127.0.0.1 is the local loopback interface on each one), I did realize while typing this update out that I had CoreDNS container running in the background 😂 it was publishing 53:53 (TCP) and 53:53/udp (UDP), but WSL2 wouldn't show the ports published via ss -tulpn like I'd usually expect to see... because that would imply *:53 for all interfaces, but WSL2 already had :53 from systemd at 127.0.0.53%lo:53 (both UDP and TCP), so Docker was like "I can't bind generically (0.0.0.0:53 / *:53)", yet this was still published and accessible in a way that the Windows 11 host could reach it....? (if I published with an explicit IP to bind the port mapping from that belonged to WSL2 that'd work and show up in ss -tulpn output, thus it was due to the port conflict)
So I bring down that CoreDNS container... voila failure:
$ ./doggo.exe TXT github.com '@127.0.0.1'
time=2024-08-07T11:46:54.035+12:00 level=ERROR msg="error in lookup" error="read udp 127.0.0.1:63598->127.0.0.1:53: i/o timeout"
time=2024-08-07T11:46:54.036+12:00 level=ERROR msg="Error looking up DNS records" error="context deadline exceeded"ipconfig /all still lists 0.0.0.0, and if I don't provide an explicit DNS server we get the expected debug failure:
./doggo.exe --debug TXT github.com
time=2024-08-07T11:47:09.949+12:00 level=DEBUG msg="LoadNameservers: Initial nameservers" nameservers=[]
time=2024-08-07T11:47:09.950+12:00 level=DEBUG msg="No user specified nameservers, falling back to system nameservers"
time=2024-08-07T11:47:09.956+12:00 level=DEBUG msg="Loaded system nameservers" nameservers="[{Address:192.168.1.1:53 Type:udp} {Address:0.0.0.0:53 Type:udp}]"
time=2024-08-07T11:47:09.956+12:00 level=DEBUG msg="initiating UDP resolver"
time=2024-08-07T11:47:09.956+12:00 level=DEBUG msg="initiating UDP resolver"
time=2024-08-07T11:47:09.956+12:00 level=DEBUG msg="Attempting to resolve" domain=github.com. ndots=0 nameserver=0.0.0.0:53
time=2024-08-07T11:47:09.956+12:00 level=DEBUG msg="Attempting to resolve" domain=github.com. ndots=0 nameserver=192.168.1.1:53
time=2024-08-07T11:47:09.965+12:00 level=DEBUG msg="Response truncated; retrying now" protocol=tcp
time=2024-08-07T11:47:09.965+12:00 level=DEBUG msg="Attempting to resolve" domain=github.com. ndots=0 nameserver=192.168.1.1:53
time=2024-08-07T11:47:12.018+12:00 level=ERROR msg="error in lookup" error="dial tcp 192.168.1.1:53: connectex: No connection could be made because the target machine actively refused it."
time=2024-08-07T11:47:14.958+12:00 level=ERROR msg="Error looking up DNS records" error="context deadline exceeded"Both were queried but neither successful 👍
It worked before with CoreDNS only because I had my local custom zones with 1.1.1.1 as a forward query for fallback.
Queries doggo vs dig
These appear to be on par now, while my original report noted a discrepancy.
I am aware of dig not using glibc IIRC? (at least that it and other DNS tools typically were more "direct", unlike other software that hooked into glibc which affected DNS resolution a bit differently, perhaps via getaddrinfo()?). In the past, Go based software with static builds when run on Alpine / musl have needed a workaround to include /etc/nsswitch.conf, otherwise the query would differ from glibc based images/distro (Alpine has been known to have various other DNS quirks too), but that has since been resolved.
TXT likewise fails with dig (NOTE: 172.18.160.1 is the DNS server in /etc/resolv.conf on WSL2, it'll match to 192.168.1.1 on the host):
$ dig TXT github.com
;; Truncated, retrying in TCP mode.
;; Connection to 172.18.160.1#53(172.18.160.1) for github.com failed: timed out.
;; no servers could be reachedAgain A record query was successful:
$ dig A github.com
; <<>> DiG 9.18.28-0ubuntu0.22.04.1-Ubuntu <<>> A github.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 37724
;; flags: qr rd ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available
;; QUESTION SECTION:
;github.com. IN A
;; ANSWER SECTION:
github.com. 0 IN A 4.237.22.38
;; Query time: 20 msec
;; SERVER: 172.18.160.1#53(172.18.160.1) (UDP)
;; WHEN: Wed Aug 07 12:21:07 NZST 2024
;; MSG SIZE rcvd: 54Reverse DNS seems to work too:
$ ./doggo -x 4.237.22.38
NAME TYPE CLASS TTL ADDRESS NAMESERVER
38.22.237.4.in-addr.arpa. PTR IN 0s github.com. 172.18.160.1:53
$ dig -x 4.237.22.38
; <<>> DiG 9.18.28-0ubuntu0.22.04.1-Ubuntu <<>> -x 4.237.22.38
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 51957
;; flags: qr rd ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; WARNING: recursion requested but not available
;; QUESTION SECTION:
;38.22.237.4.in-addr.arpa. IN PTR
;; ANSWER SECTION:
38.22.237.4.in-addr.arpa. 0 IN PTR github.com.
;; Query time: 0 msec
;; SERVER: 172.18.160.1#53(172.18.160.1) (UDP)
;; WHEN: Wed Aug 07 12:24:23 NZST 2024
;; MSG SIZE rcvd: 90Docker
Although not with doggo in the Docker container:
$ docker run --rm -v ./doggo:/opt/doggo:ro --workdir /opt alpine ash -c './doggo --debug -x 4.237.22.38'
time=2024-08-07T00:29:42.802Z level=DEBUG msg="LoadNameservers: Initial nameservers" nameservers=[]
time=2024-08-07T00:29:42.802Z level=DEBUG msg="No user specified nameservers, falling back to system nameservers"
time=2024-08-07T00:29:42.802Z level=DEBUG msg="Loaded system nameservers" nameservers="[{Address:192.168.65.7:53 Type:udp}]"
time=2024-08-07T00:29:42.802Z level=DEBUG msg="initiating UDP resolver"
time=2024-08-07T00:29:42.802Z level=DEBUG msg="Attempting to resolve" domain=38.22.237.4.in-addr.arpa. ndots=0 nameserver=192.168.65.7:53
NAME TYPE CLASS TTL ADDRESS NAMESERVER$ docker run --rm -v ./doggo:/opt/doggo:ro --workdir /opt alpine ash -c 'apk add bind-tools && dig -x 4.237.22.38'
; <<>> DiG 9.18.24 <<>> -x 4.237.22.38
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 20531
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;38.22.237.4.in-addr.arpa. IN PTR
;; Query time: 1050 msec
;; SERVER: 192.168.65.7#53(192.168.65.7) (UDP)
;; WHEN: Wed Aug 07 00:27:48 UTC 2024
;; MSG SIZE rcvd: 42Now with Cloudflare 1.1.1.1 as DNS:
$ docker run --rm -v ./doggo:/opt/doggo:ro --workdir /opt alpine ash -c './doggo --debug -x 4.237.22.38 @1.1.1.1'
time=2024-08-07T00:30:54.542Z level=DEBUG msg="LoadNameservers: Initial nameservers" nameservers=[1.1.1.1]
time=2024-08-07T00:30:54.542Z level=DEBUG msg="Added nameserver" nameserver="{Address:1.1.1.1:53 Type:udp}"
time=2024-08-07T00:30:54.542Z level=DEBUG msg="LoadNameservers: Final nameservers" nameservers="[{Address:1.1.1.1:53 Type:udp}]"
time=2024-08-07T00:30:54.542Z level=DEBUG msg="initiating UDP resolver"
time=2024-08-07T00:30:54.543Z level=DEBUG msg="Attempting to resolve" domain=38.22.237.4.in-addr.arpa. ndots=0 nameserver=1.1.1.1:53
NAME TYPE CLASS TTL ADDRESS NAMESERVER STATUS
22.237.4.in-addr.arpa. SOA IN 156s ns1-04.azure-dns.com. 1.1.1.1:53 NXDOMAIN
azuredns-hostmaster.microsoft.com.
1 3600 300 2419200 300$ docker run --rm -v ./doggo:/opt/doggo:ro --workdir /opt alpine ash -c 'apk add bind-tools && dig -x 4.237.22.38 @1.1.1.1'
; <<>> DiG 9.18.24 <<>> -x 4.237.22.38 @1.1.1.1
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 47403
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;38.22.237.4.in-addr.arpa. IN PTR
;; AUTHORITY SECTION:
22.237.4.in-addr.arpa. 125 IN SOA ns1-04.azure-dns.com. azuredns-hostmaster.microsoft.com. 1 3600 300 2419200 300
;; Query time: 10 msec
;; SERVER: 1.1.1.1#53(1.1.1.1) (UDP)
;; WHEN: Wed Aug 07 00:31:26 UTC 2024
;; MSG SIZE rcvd: 139That seems to suggest that dig got a cached response on WSL2 for the reverse DNS mapping to Github since that was resolved earlier?
WSL2
Sure enough when I queried back on WSL2 we got the response that we'd get from 1.1.1.1:
$ ./doggo -x 4.237.22.38
NAME TYPE CLASS TTL ADDRESS NAMESERVER STATUS
22.237.4.in-addr.arpa. SOA IN 299s ns1-04.azure-dns.com. 172.18.160.1:53 NXDOMAIN
azuredns-hostmaster.microsoft.com.
1 3600 300 2419200 300
$ ./doggo -x 4.237.22.38 @1.1.1.1
NAME TYPE CLASS TTL ADDRESS NAMESERVER STATUS
22.237.4.in-addr.arpa. SOA IN 57s ns1-04.azure-dns.com. 1.1.1.1:53 NXDOMAIN
azuredns-hostmaster.microsoft.com.
1 3600 300 2419200 300By performing a query for the A record like had been done earlier in this report, that affects the local resolution:
$ ./doggo A github.com
NAME TYPE CLASS TTL ADDRESS NAMESERVER
github.com. A IN 0s 4.237.22.38 172.18.160.1:53
$ ./doggo -x 4.237.22.38
NAME TYPE CLASS TTL ADDRESS NAMESERVER
38.22.237.4.in-addr.arpa. PTR IN 0s github.com. 172.18.160.1:53
# But as expected 1.1.1.1 is unaffected:
$ ./doggo -x 4.237.22.38 @1.1.1.1
NAME TYPE CLASS TTL ADDRESS NAMESERVER STATUS
22.237.4.in-addr.arpa. SOA IN 280s ns1-04.azure-dns.com. 1.1.1.1:53 NXDOMAIN
azuredns-hostmaster.microsoft.com.
1 3600 300 2419200 300That doesn't change in the container with Dockers embedded DNS server (192.168.65.7), thus the WSL2 (172.18.160.1) is responding differently by temporarily caching the IP to github.com.
172.18.160.1 is managed by WSL2 adding to /etc/resolv.conf, while the Ubuntu WSL2 install it's interface assigned to an IP in the 172.18.0.0/20 subnet (range: 172.18.160.1 - 172.18.175.254). So I assume the caching is going on there and differs from Windows 11 which connects directly to (EDIT: Nope seems to be cached at 192.168.1.1 👍192.168.1.1)
Windows 11 host
The Windows 11 host likewise behaves similarly (with the additional error for 192.168.1.1 which represents the actual router device handling the DNS query for each host/wsl2/docker client):
$ ./doggo.exe --debug -x 4.237.22.38
time=2024-08-07T12:52:23.652+12:00 level=DEBUG msg="LoadNameservers: Initial nameservers" nameservers=[]
time=2024-08-07T12:52:23.653+12:00 level=DEBUG msg="No user specified nameservers, falling back to system nameservers"
time=2024-08-07T12:52:23.660+12:00 level=DEBUG msg="Loaded system nameservers" nameservers="[{Address:192.168.1.1:53 Type:udp} {Address:0.0.0.0:53 Type:udp}]"
time=2024-08-07T12:52:23.660+12:00 level=DEBUG msg="initiating UDP resolver"
time=2024-08-07T12:52:23.660+12:00 level=DEBUG msg="initiating UDP resolver"
time=2024-08-07T12:52:23.660+12:00 level=DEBUG msg="Attempting to resolve" domain=38.22.237.4.in-addr.arpa. ndots=0 nameserver=0.0.0.0:53
time=2024-08-07T12:52:23.660+12:00 level=DEBUG msg="Attempting to resolve" domain=38.22.237.4.in-addr.arpa. ndots=0 nameserver=192.168.1.1:53
time=2024-08-07T12:52:28.666+12:00 level=ERROR msg="Error looking up DNS records" error="context deadline exceeded"But it can't seem to resolve the A record either:
$ ./doggo.exe --debug A github.com
time=2024-08-07T12:54:54.242+12:00 level=DEBUG msg="LoadNameservers: Initial nameservers" nameservers=[]
time=2024-08-07T12:54:54.243+12:00 level=DEBUG msg="No user specified nameservers, falling back to system nameservers"
time=2024-08-07T12:54:54.248+12:00 level=DEBUG msg="Loaded system nameservers" nameservers="[{Address:192.168.1.1:53 Type:udp} {Address:0.0.0.0:53 Type:udp}]"
time=2024-08-07T12:54:54.248+12:00 level=DEBUG msg="initiating UDP resolver"
time=2024-08-07T12:54:54.248+12:00 level=DEBUG msg="initiating UDP resolver"
time=2024-08-07T12:54:54.249+12:00 level=DEBUG msg="Attempting to resolve" domain=github.com. ndots=0 nameserver=0.0.0.0:53
time=2024-08-07T12:54:54.249+12:00 level=DEBUG msg="Attempting to resolve" domain=github.com. ndots=0 nameserver=192.168.1.1:53
time=2024-08-07T12:54:59.260+12:00 level=ERROR msg="Error looking up DNS records" error="context deadline exceeded"Presumably the 0.0.0.0 query (which no longer has any other DNS server to connect to with my CoreDNS container no longer running) is causing that failure?
By providing the router DNS nameserver explicitly we actually get a result returned:
$ ./doggo.exe --debug -x 4.237.22.38 '@192.168.1.1'
time=2024-08-07T12:58:41.878+12:00 level=DEBUG msg="LoadNameservers: Initial nameservers" nameservers=[192.168.1.1]
time=2024-08-07T12:58:41.878+12:00 level=DEBUG msg="Added nameserver" nameserver="{Address:192.168.1.1:53 Type:udp}"
time=2024-08-07T12:58:41.878+12:00 level=DEBUG msg="LoadNameservers: Final nameservers" nameservers="[{Address:192.168.1.1:53 Type:udp}]"
time=2024-08-07T12:58:41.878+12:00 level=DEBUG msg="initiating UDP resolver"
time=2024-08-07T12:58:41.878+12:00 level=DEBUG msg="Attempting to resolve" domain=38.22.237.4.in-addr.arpa. ndots=0 nameserver=192.168.1.1:53
NAME TYPE CLASS TTL ADDRESS NAMESERVER STATUS
22.237.4.in-addr.arpa. SOA IN 300s ns1-04.azure-dns.com. 192.168.1.1:53 NXDOMAIN
azuredns-hostmaster.microsoft.com.
1 3600 300 2419200 300$ ./doggo.exe --debug A github.com '@192.168.1.1'
time=2024-08-07T12:56:14.792+12:00 level=DEBUG msg="LoadNameservers: Initial nameservers" nameservers=[192.168.1.1]
time=2024-08-07T12:56:14.793+12:00 level=DEBUG msg="Added nameserver" nameserver="{Address:192.168.1.1:53 Type:udp}"
time=2024-08-07T12:56:14.793+12:00 level=DEBUG msg="LoadNameservers: Final nameservers" nameservers="[{Address:192.168.1.1:53 Type:udp}]"
time=2024-08-07T12:56:14.793+12:00 level=DEBUG msg="initiating UDP resolver"
time=2024-08-07T12:56:14.794+12:00 level=DEBUG msg="Attempting to resolve" domain=github.com. ndots=0 nameserver=192.168.1.1:53
NAME TYPE CLASS TTL ADDRESS NAMESERVER
github.com. A IN 34s 4.237.22.38 192.168.1.1:53That shares the same observation of caching A for reverse DNS queries that was observed with WSL2, so that's all happening at the router 192.168.1.1, not Windows host or WSL2 own nameserver IP.
Just sharing this error I came across (although it seems to be dependent on nameserver used for the query):
I'm aware of the local DNS service being a bit faulty (see details below), I'm not sure if this is something
doggocan actually resolve either (affectsqtoo, so I suppose both projects are handling the query in the same way?).digcan resolve the record without issue in the same environment, thus this failure may be specific to Go (or a common DNS package, I haven't comparedqanddoggopackages or src).As can be seen below,
github.comhas a largeTXTrecord that splits into multiple parts, I suspect that's the one related to the overflow error?qhas a option--txtconcatto merge these into a single string for displayed output, but the error seems to suggest that this is a failure at the DNS name server used for the query not being able to handle this.My local DNS in this case is a bit complicated
/etc/resolv.confmanaged by Docker Desktop (192.168.65.0/24network configured, Docker manages it's own internal DNS layer)./etc/resolv.conf(172.x.y.znameserver), but Docker Desktop may actually run containers in a separate VM IIRC, rather than this Ubuntu 22.04 one that is provisioned by default and where I interface with the container via Windows Terminal. So this name server probably isn't in use, I know that if the container runs in host mode networking it's not reachable from my WSL2 Ubuntu 22.04 or Windows 11 host.1.1.1.1configured for the nameserver to use, DNS should be going through this to reach the internet AFAIK, so it may be an issue with Dockers internally managed DNS layer and potentially whatever upstream it routes queries through.dig/drillalso confirm that I cannot perform reverse DNS queries either, unless I provide an alternative name server to query like via@1.1.1.1.@1.1.1.1, but Docker containers do not return anything. I've also changed the Windows host network to use DHCP again instead of1.1.1.1and there is no issue there either for Ubuntu.dighas no trouble in a container however querying theTXTrecord forgithub.com.Here is the containers
/etc/resolv.conf(comment is generated from the managed file contents by Docker):While the Windows host is configured with
1.1.1.1, it wasn't when these issues started and the Docker daemon has not been restarted yet. It's also possible that the comment is misleading if the Docker host in this case is the WSL2 VM that Docker Desktop manages, where it may set a different nameserver (not sure how to access that). So I suspect either the issue is temporary, or the internal DNS layer Docker has is faulty.Via DNS server in
/etc/resolv.conf(fail)For reference, same failure via
q:Via
@1.1.1.1(success):