Closed jermudgeon closed 1 year ago
This seems to still be the case with macOS version 12.5.1
Similar position here.
These work:
dig @172.22.xx.xx myhost.xz
nslookup myhost.xz 172.22.xx.xx
But dig myhost.xz
and nslookup myhost.xz
do not 🤔
I have the DNS settings in zerotier configured with a search domain of "xz" and a server IP of 172.22.xx.xx
Very puzzling, it's like everything is there but somehow zerotier is having trouble convincing macOS that it should be involved in hostname resolution
dig, host, nslookup,... won't work on macos for this. DNS queries go through some other system.
try dns-sd -G v4 myhost.xz
, dscacheutil -q host -a name
, or use ping
Hello!
This boils down to which DNS libraries a utility uses for name resolution. Some OSX commands, like ping, will pick up the per-interface name resolution. Others, like the bind9 utils (dig, nslookup), do not use the proper APIs, and will fail to resolve.
Try variations of these commands:
scutil --dns
scutil -W -r my.internal.name
PS: This is firmly a MacOS issue... it falls well outside the scope of ZeroTier (or ZeroNSD).
-s
Interesting! The only thing that remains confusing to me is I tried to reach these hosts over ssh and via a web browser and couldn’t, I only used dig/nslookup after the fact to try to debug.
I could reach them in the past, and trying again today I can now, so I know I had things configured correctly. What I can’t figure out is why it doesn’t work sometimes.
I’ll use the cli tools you’ve suggested next time to debug instead of dig, maybe that will provide an interesting clue.
Update: I’ve been unable to reproduce this since and I feel quite confident I made a mistake somewhere and ZeroTier was not the issue. Thanks for the tips on properly testing this!
I'm still having the original issue. @someara My issue was not with dig/host/nslookup; it was with tools like ping and ssh, which should use system-wide resolvers.
Steps to reproduce: Ventura 13.01 ZT 1.10.2
Targeted lookups work just fine (dig @), so there is no underlying ZT configuration problem.
There is also no underlying resolver problem; I can manually add the resolver:
echo 'nameserver <ip>' > /etc/resolver/<my.domain>
So it still appears to me that the ZT client is not correctly updating the resolver configuration.
macos 13.4.1 (c),ZeroTier 1.10.6. same issue
macOS 13.4.1 (c) Zerotier 1.10.6 same issue here as well
How are you testing? We can't reproduce this.
Nslookup for custom domain isn’t working on mac but works on any other platform with zt installed. By the way this issue happened after recent apple macos patch. Toggling “allow dns configuration” isn’t changing anything. scutil --dns show no chances ether
Working for me (OP) on Ventura 13.4.1 and ZT 1.10.6 — nmap, ping, and Safari all resolve correctly, at least for FQDN
clean install, enable dns, zero resolution, nothing in scutil
if I manually add to scutil everything works fine. IMO it is a zerotier problem
unless there is a quirk on how the internal dns has to be specified in the portal. it absolutely does not work. all windows clients work fine.
I believe you, I have definitely had times in the past where scutil --dns
looks correct. For example, right now I get
resolver #9 domain : <removed> nameserver[0] : <removed> flags : Request A records, Request AAAA records reach : 0x00020002 (Reachable,Directly Reachable Address)
I know nslookup is avoiding dns interface settings. but ping alway use correct resolver. Since mac upgrade ZT just stop resolving. I know it might be related that all-weird apple private relay, but the problem is here. I was using ZT for the past 3 years and won't make the bug report if the problem isn't here. When I do nslookup machine.example.com ip-domain-resolver - it gets correct ip but ping machine.example.com - ping: cannot resolve machine.example.com: Unknown host That's ZT issue as I see it
"I believe you, I have definitely had times in the past where scutil --dns looks correct." for me, it does not have a entry for the dns name at all.
go in scutil
and do
list
show State:/Network/Service/<network-id>/DNS
it should look something like this
<dictionary> {
ServerAddresses : <array> {
0 : fdb1:xxxx:912e:7339:699:938f:5e69:b6a
1 : 10.123.2.1
}
SupplementalMatchDomains : <array> {
0 : my.domain
}
}
I was on 13.0 (new mac) and now I'm on 13.4.1. They both seem to work. Would love to get to the bottom of this.
You can post the output of zerotier-cli listnetworks -j
, obscure your network ID if you want.
zerotier-cli listnetworks -j
[ { "allowDNS": true, "allowDefault": false, "allowGlobal": false, "allowManaged": true, "assignedAddresses": [ "fd1d:7193:9404:bded:1c99:935e:b824:7a5/88", "172.20.30.253/24" ], "bridge": false, "broadcastEnabled": true, "dhcp": false, "dns": { "domain": "i.domain.net.au", "servers": [ "172.20.30.2", "172.20.30.3", "172.20.30.5" ] }, "id": "ffffffffffffffff", "mac": "1e:b3:05:20:93:36", "mtu": 2800, "multicastSubscriptions": [ { "adi": 0, "mac": "01:00:5e:00:00:01" }, { "adi": 0, "mac": "01:00:5e:00:00:fb" }, { "adi": 0, "mac": "33:33:00:00:00:01" }, { "adi": 0, "mac": "33:33:00:00:00:fb" }, { "adi": 0, "mac": "33:33:ff:14:87:23" }, { "adi": 0, "mac": "33:33:ff:20:93:36" }, { "adi": 0, "mac": "33:33:ff:24:07:a5" }, { "adi": 2886999805, "mac": "ff:ff:ff:ff:ff:ff" } ], "name": "domain.net.au-core", "netconfRevision": 14, "nwid": "ffffffffffffffff", "portDeviceName": "feth1089", "portError": 0, "routes": [ { "flags": 0, "metric": 0, "target": "172.18.0.0/16", "via": "172.20.30.1" }, { "flags": 0, "metric": 0, "target": "172.20.1.0/24", "via": "172.20.30.1" }, { "flags": 0, "metric": 0, "target": "172.20.30.0/24", "via": null }, { "flags": 0, "metric": 0, "target": "2001:8ffff:ffff:ff::/64", "via": "fd1d:7193:9404:bded:1c99:93f9:1f43:db55" } ], "status": "OK", "type": "PRIVATE" } ]
`show State:/Network/Service/ffffffffffffffff/DNS
> show State:/Network/Service/xxxxxxxx/DNS
<dictionary> {
ServerAddresses : <array> {
0 : 172.24.0.10
1 : 172.24.0.1
}
SupplementalMatchDomains : <array> {
0 : myrealdomain.ca
}
}
zerotier-cli listnetworks -j
[
{
"allowDNS": true,
"allowDefault": false,
"allowGlobal": false,
"allowManaged": true,
"assignedAddresses": [
"fd6a:b565:387a:b525:1199:9392:85d5:6346/88",
"172.24.244.201/16"
],
"authenticationExpiryTime": 0,
"authenticationURL": "",
"bridge": false,
"broadcastEnabled": false,
"dhcp": false,
"dns": {
"domain": "myrealdomain.ca",
"servers": [
"172.24.0.10",
"172.24.0.1"
]
},
"id": "6ab565387ab52511",
"mac": "12:b7:30:af:5b:23",
"mtu": 2800,
"multicastSubscriptions": [
{
"adi": 0,
"mac": "01:00:5e:00:00:01"
},
{
"adi": 0,
"mac": "01:00:5e:00:00:fb"
},
{
"adi": 0,
"mac": "33:33:00:00:00:01"
},
{
"adi": 0,
"mac": "33:33:00:00:00:fb"
},
{
"adi": 0,
"mac": "33:33:ff:af:5b:23"
},
{
"adi": 0,
"mac": "33:33:ff:d1:65:ff"
},
{
"adi": 0,
"mac": "33:33:ff:d5:63:46"
},
{
"adi": 2887316681,
"mac": "ff:ff:ff:ff:ff:ff"
}
],
"name": "The Name",
"netconfRevision": 184,
"nwid": "xxxxxxxxx",
"portDeviceName": "feth2668",
"portError": 0,
"routes": [
{
"flags": 0,
"metric": 0,
"target": "104.18.2.147/32",
"via": "172.24.0.1"
},
{
"flags": 0,
"metric": 0,
"target": "104.18.3.147/32",
"via": "172.24.0.1"
},
{
"flags": 0,
"metric": 0,
"target": "162.159.136.70/32",
"via": "172.24.0.1"
},
{
"flags": 0,
"metric": 0,
"target": "162.159.137.70/32",
"via": "172.24.0.1"
},
{
"flags": 0,
"metric": 0,
"target": "172.20.0.0/21",
"via": "172.24.0.1"
},
{
"flags": 0,
"metric": 0,
"target": "172.21.0.0/21",
"via": "172.24.0.2"
},
{
"flags": 0,
"metric": 0,
"target": "172.24.0.0/16",
"via": null
},
{
"flags": 0,
"metric": 0,
"target": "52.224.196.54/32",
"via": "172.24.0.1"
},
{
"flags": 0,
"metric": 0,
"target": "78.25.196.229/32",
"via": "172.24.0.1"
}
],
"ssoEnabled": true,
"status": "OK",
"type": "PRIVATE"
}
]
scutil --dns
DNS configuration
resolver #1
search domain[0] : local
nameserver[0] : 2604:3d09:6b80:1882::303
nameserver[1] : fe80::4da5:1f53:b778:d26d%14d
nameserver[2] : 192.168.1.2
nameserver[3] : 192.168.1.4
if_index : 14 (en0)
flags : Request A records, Request AAAA records
reach : 0x00020002 (Reachable,Directly Reachable Address)
resolver #2
domain : local
options : mdns
timeout : 5
flags : Request A records, Request AAAA records
reach : 0x00000000 (Not Reachable)
order : 300000
resolver #3
domain : 254.169.in-addr.arpa
options : mdns
timeout : 5
flags : Request A records, Request AAAA records
reach : 0x00000000 (Not Reachable)
order : 300200
resolver #4
domain : 8.e.f.ip6.arpa
options : mdns
timeout : 5
flags : Request A records, Request AAAA records
reach : 0x00000000 (Not Reachable)
order : 300400
resolver #5
domain : 9.e.f.ip6.arpa
options : mdns
timeout : 5
flags : Request A records, Request AAAA records
reach : 0x00000000 (Not Reachable)
order : 300600
resolver #6
domain : a.e.f.ip6.arpa
options : mdns
timeout : 5
flags : Request A records, Request AAAA records
reach : 0x00000000 (Not Reachable)
order : 300800
resolver #7
domain : b.e.f.ip6.arpa
options : mdns
timeout : 5
flags : Request A records, Request AAAA records
reach : 0x00000000 (Not Reachable)
order : 301000
DNS configuration (for scoped queries)
resolver #1
search domain[0] : local
nameserver[0] : 2604:3d09:6b80:1882::303
nameserver[1] : fe80::4da5:1f53:b778:d26d%14d
nameserver[2] : 192.168.1.2
nameserver[3] : 192.168.1.4
if_index : 14 (en0)
flags : Scoped, Request A records, Request AAAA records
reach : 0x00020002 (Reachable,Directly Reachable Address)
P.S. My default settings for dhcp in my lan is ipv6
I'm finding similar reports for ventura on vpn app/projects. Can't find any solutions.
If you change your config so it uses just 1 ipv4 address in the dns server list, does it work? That's the only difference I can see between my configs. I don't have a good way to setup two servers at the moment.
Random reddit post says if any of your resolvers support DNSSEC, it will ignore any resolvers that don't have it.
Ok, ive removed the manual dns forwarder,
scobber@Scotts-MacBook-Pro ~ % scutil --dns
DNS configuration
resolver #1
search domain[0] : wifi.local
nameserver[0] : 0000:0000:0000:1::1
nameserver[1] : 192.168.5.1
if_index : 13 (en0)
flags : Request A records, Request AAAA records
reach : 0x00020002 (Reachable,Directly Reachable Address)
resolver #2
domain : local
options : mdns
timeout : 5
flags : Request A records, Request AAAA records
reach : 0x00000000 (Not Reachable)
order : 300000
resolver #3
domain : 254.169.in-addr.arpa
options : mdns
timeout : 5
flags : Request A records, Request AAAA records
reach : 0x00000000 (Not Reachable)
order : 300200
resolver #4
domain : 8.e.f.ip6.arpa
options : mdns
timeout : 5
flags : Request A records, Request AAAA records
reach : 0x00000000 (Not Reachable)
order : 300400
resolver #5
domain : 9.e.f.ip6.arpa
options : mdns
timeout : 5
flags : Request A records, Request AAAA records
reach : 0x00000000 (Not Reachable)
order : 300600
resolver #6
domain : a.e.f.ip6.arpa
options : mdns
timeout : 5
flags : Request A records, Request AAAA records
reach : 0x00000000 (Not Reachable)
order : 300800
resolver #7
domain : b.e.f.ip6.arpa
options : mdns
timeout : 5
flags : Request A records, Request AAAA records
reach : 0x00000000 (Not Reachable)
order : 301000
DNS configuration (for scoped queries)
resolver #1
search domain[0] : wifi.local
nameserver[0] : 2001:8000:2ee0:1::1
nameserver[1] : 192.168.5.1
if_index : 13 (en0)
flags : Scoped, Request A records, Request AAAA records
reach : 0x00020002 (Reachable,Directly Reachable Address)
scobber@Scotts-MacBook-Pro ~ % zerotier-cli listnetworks -j
[
{
"allowDNS": true,
"allowDefault": false,
"allowGlobal": false,
"allowManaged": true,
"assignedAddresses": [
"fd1d:7193:9404:bded:1c99:935e:b824:7a5/88",
"172.20.30.253/24"
],
"bridge": false,
"broadcastEnabled": true,
"dhcp": false,
"dns": {
"domain": "i.domain",
"servers": [
"172.20.30.2"
]
},
"id": "ffffffffffffffff",
"mac": "1e:b3:05:20:93:36",
"mtu": 2800,
"multicastSubscriptions": [
{
"adi": 0,
"mac": "01:00:5e:00:00:01"
},
{
"adi": 0,
"mac": "01:00:5e:00:00:fb"
},
{
"adi": 0,
"mac": "33:33:00:00:00:01"
},
{
"adi": 0,
"mac": "33:33:00:00:00:fb"
},
{
"adi": 0,
"mac": "33:33:ff:20:93:36"
},
{
"adi": 0,
"mac": "33:33:ff:24:07:a5"
},
{
"adi": 0,
"mac": "33:33:ff:bd:a7:83"
},
{
"adi": 2886999805,
"mac": "ff:ff:ff:ff:ff:ff"
}
],
"name": "domain-core",
"netconfRevision": 18,
"nwid": "ffffffffffffffff",
"portDeviceName": "feth1089",
"portError": 0,
"routes": [
{
"flags": 0,
"metric": 0,
"target": "172.18.0.0/16",
"via": "172.20.30.1"
},
{
"flags": 0,
"metric": 0,
"target": "172.20.1.0/24",
"via": "172.20.30.1"
},
{
"flags": 0,
"metric": 0,
"target": "172.20.30.0/24",
"via": null
},
{
"flags": 0,
"metric": 0,
"target": "0000:0000:0000:3::/64",
"via": "fd1d:7193:9404:bded:1c99:93f9:1f43:db55"
}
],
"status": "OK",
"type": "PRIVATE"
}
]
still no luck, still have routing by ip working, dns server is alive
scobber@Scotts-MacBook-Pro ~ % nslookup [www.google.com](http://www.google.com/) 172.20.30.2
Server: 172.20.30.2
Address: 172.20.30.2#53
Non-authoritative answer:
Name: [www.google.com](http://www.google.com/)
Address: 142.250.70.196
scobber@Scotts-MacBook-Pro ~ % nslookup i.domain 172.20.30.2
Server: 172.20.30.2
Address: 172.20.30.2#53
Name: i.domain
Address: 172.20.30.2
Name: i.domain
Address: 172.16.0.5
Name: i.domain
Address: 172.20.1.2
the dns servers here don't do DNSSEC either.
further testing adding manual entries again
scobber@Scotts-MacBook-Pro ~ % ping i.domain
ping: cannot resolve i.domain: Unknown host
scobber@Scotts-MacBook-Pro ~ % ./installdns.sh
f.read: reading file (dns.txt).
1> d.init
1> d.add ServerAddresses * 172.20.30.2
1> d.add SupplementalMatchDomains * i.domain
1> set State:/Network/Service/idomain/DNS
f.read: reading file (otherdns.txt).
1> d.init
1> d.add ServerAddresses * 172.20.30.2
1> d.add SupplementalMatchDomains * office.otherdomain
1> set State:/Network/Service/officeotherdomain/DNS
scobber@Scotts-MacBook-Pro ~ % ping i.domain
PING i.domain (172.20.30.2): 56 data bytes
64 bytes from 172.20.30.2: icmp_seq=0 ttl=128 time=5.537 ms
64 bytes from 172.20.30.2: icmp_seq=1 ttl=128 time=27.101 ms
64 bytes from 172.20.30.2: icmp_seq=2 ttl=128 time=11.639 ms
64 bytes from 172.20.30.2: icmp_seq=3 ttl=128 time=7.630 ms
^C
--- i.domain ping statistics ---
4 packets transmitted, 4 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 5.537/12.977/27.101/8.444 ms
thanks for testing. what's in your show State:/Network/Service/ffffffffffffffff/DNS
? It should be the same as your manual test.
Does a leave and rejoin help?
yours doesn't show even in the main (not scoped) resolver list. I have
resolver #3
domain : home.arpa
nameserver[0] : 10.243.51.1
flags : Supplemental, Request A records, Request AAAA records
reach : 0x00020002 (Reachable,Directly Reachable Address)
order : 102600
leave / rejoin does not help
show State:/Network/Service/ffffffffffffffff/DNS
<dictionary> {
ServerAddresses : <array> {
0 : 172.20.30.2
}
SupplementalMatchDomains : <array> {
0 : i.domain
}
}
thanks. no idea why it doesn't work. it's exactly the same as what you did manually with scutil.
the only thing that is different about the output of the both, is if it is manually created, it appears in the scutil --dns list, where if its created by ZT it does not.
its not some sort of annoying gatekeeper thing?
I'm not sure. If you figure out what makes it work for some networks and not others let us know! Or what makes it work on some computers and not others.
I think I can reproduce. It doesn't work when there is no ipv6 dns server defined.
scutil --dns only shows my v6 nameservers if there are no v4 name servers, it doesn't show the resolver at all
will work on a fix. in the meantime, enable rfc4193 addresses with the check box on central. zeronsd will automatically use them.
In my case ipv6 dns servers are defined, together with ipv4 https://github.com/zerotier/ZeroTierOne/issues/1696#issuecomment-1646713685
@myevit for clarity, are you routing the v6 subnet via ZT, and if its a public range do you need to have allow global on thinking it could be getting dropped by your internet facing perimeter firewall
@myevit for clarity, are you routing the v6 subnet via ZT, and if its a public range do you need to have allow global on thinking it could be getting dropped by your internet facing perimeter firewall
I do not route ipv6 into ZT, network clients are using only ipv4. My macbook is sitting in ipv6+ipv4 LAN. Same LAN where I have my Windows PC. On Windows interface DNS lookups are working correctly, on macbook not.
my way of thinking is similar, I don't assign v6 dns, however if a lookup results in a v6 response I route to the internal /48. I can keep the edge pretty firm and maintain a v6 dmz. I don't particularly want to assign v6 resolvers even though they are there, prioritising v4 keeps things simple, but routing v6 via zt limits domain traffic escaping via the internet
@Scobber Out of curiosity, do you have installed zerotierNSD? https://github.com/zerotier/zeronsd I do use it
Nah I don't use it, and for my implementation I don't think it would be helpful. I have ms dns setup, and have reverse dns working across the zerotier ip scope. 99% of my network is off zerotier, I have a couple of deb boxes that serve as gateways. I have written some middleware for windows that will join/dejoin windows machines based on the environment around them, so zt is only ever active when a device is away from the business wan. so they are not always connected when on the wan. The MacBook is my own computer, mac's definitely have their niche. it's like using a linux machine with a complete interface. although definitely not the same.
I see. I though it might me related, as nsd is talking with api or something. I have completely made overlay network for all devices, totally unnecessary but it was easier then make scripts, and make scripts work. Also I have replaced domain controller with zerotierNSD and Azure AD.
Aside note I have never made ZT clients ping6/talk to each other via ipv6, I wonder if it's related with the issue. Also my LAN devices get ipv6 from /64 delegated ip and my with custom inLAN DNS filter/server ipv6 address with is also "public" from /64 delegation. Maybe ipv6 resolver outweight ipv4 in new macs? Or there is mac bug that not allowing zt put ipv4 record with zt resolver if ipv6 is active, and zt never put ipv6 resolver? Just thinking out loud.
Ventura won't use a ipv4 resolver set by zerotier. There is a fix in. See commit links above. Not sure when the next release will be.
Working in macOS 13.5.2, ZT 1.12.2 thank you for your help.
Thanks!
For fun, try the search domains: ping somehost
(without the .example.com
)
Thanks!
For fun, try the search domains:
ping somehost
(without the.example.com
)
Search domains are working too, thank you
ZT 1.10.0 macOS 12.4
Behavior: setting domain and DNS servers on a network does not successfully apply to the OS Steps to replicate:
Configure and confirm that ZT network has DNS settings enabled on client, including leave/rejoin: `zerotier-cli listnetworks -j