DNS not working after installing netbird 0.28.2 on Ubuntu

[tomasznguyen]

Describe the problem

After a fresh install of Ubuntu (24.04), resolving domains works. However, after installing netbird, DNS resolving does not work anymore. For example:

root@fubar:~/netbird# nslookup google.com
;; communications error to 127.0.0.53#53: timed out
;; communications error to 127.0.0.53#53: timed out
;; communications error to 127.0.0.53#53: timed out
;; no servers could be reached

root@fubar:~/netbird# dig google.com
;; communications error to 127.0.0.53#53: timed out
;; communications error to 127.0.0.53#53: timed out
;; communications error to 127.0.0.53#53: timed out

; <<>> DiG 9.18.24-0ubuntu5-Ubuntu <<>> google.com
;; global options: +cmd
;; no servers could be reached

root@fubar:~/netbird# netbird status -dA
Peers detail:
 oneplus7t-eea.netbird.selfhosted:
  NetBird IP: 100.99.169.30/32
  Public key: bTdFb8ukmy5ukjXe/MwJKdGp80CIz+E/m3l/ermnShA=
  Status: Disconnected
  -- detail --
  Connection type: P2P
  Direct: false
  ICE candidate (Local/Remote): host/prflx
  ICE candidate endpoints (Local/Remote): 10.18.0.12:51820/198.51.100.0:51820
  Last connection update: 14 minutes, 36 seconds ago
  Last WireGuard handshake: 2 minutes ago
  Transfer status (received/sent) 2.3 KiB/888 B
  Quantum resistance: false
  Routes: -
  Latency: 0s

 macbook-pro-2.netbird.selfhosted:
  NetBird IP: 100.99.229.225
  Public key: bUcXgQz0McQXxDDI0LVyq0KlVNkq6rQ0KElZ6kMTKQQ=
  Status: Connected
  -- detail --
  Connection type: P2P
  Direct: true
  ICE candidate (Local/Remote): host/prflx
  ICE candidate endpoints (Local/Remote): 10.18.0.12:51820/198.51.100.0:51820
  Last connection update: 14 minutes, 33 seconds ago
  Last WireGuard handshake: 2 minutes ago
  Transfer status (received/sent) 2.3 KiB/888 B
  Quantum resistance: false
  Routes: -
  Latency: 5.890435ms

OS: linux/amd64
Daemon version: 0.28.2
CLI version: 0.28.2
Management: Connected to https://vpn.anon-2DdBv.domain:443
Signal: Connected to https://vpn.anon-2DdBv.domain:443
Relays: 
  [stun:vpn.anon-2DdBv.domain:3478] is Unavailable, reason: dial: failed to listen: dial: dial udp: lookup vpn.anon-2DdBv.domain on 127.0.0.53:53: read udp 127.0.0.1:34617->127.0.0.53:53: i/o timeout
  [turn:vpn.anon-2DdBv.domain:3478?transport=udp] is Unavailable, reason: create client: lookup vpn.anon-2DdBv.domain on 127.0.0.53:53: read udp 127.0.0.1:46605->127.0.0.53:53: i/o timeout
Nameservers: 
  [8.8.8.8:53, 8.8.4.4:53] for [.] is Available
FQDN: frank-vpn-peer.netbird.selfhosted
NetBird IP: 100.99.227.52/16
Interface type: Kernel
Quantum resistance: false
Routes: 198.51.100.1/32, 198.51.100.2/32, 198.51.100.3/32, 198.51.100.4/32, 198.51.100.5/32, 198.51.100.6/32, anon-axyCw.domain
Peers count: 1/2 Connected

To Reproduce

Steps to reproduce the behavior:

1. Freshly install Ubuntu
2. Run ping google.com
3. Install netbird: curl -fsSL https://pkgs.netbird.io/install.sh | sh
4. Run netbird: netbird up --management-url xxx --setup-key xxx
5. Run again ping google.com

Expected behavior

DNS should be still working after installing and running netbird.

Are you using NetBird Cloud?

Self-hosted

NetBird version

0.28.2

NetBird status -d output:

See above

[svardie]

I have same problem.

[mlsmaycon]

Hello Folks,

There is a bug we introduced with the latest release where the new firewall rules are affecting the local traffic from the systemd-resolved;

if you are using exit nodes or DNS routes, you need to downgrade the routing peer to 0.27.10. If you only have DNS routes, you will need to downgrade too, but you will need to add a temporary exit node route, with a distribution group without peers. This will make the routing peer create the necessary local firewall rules.

We are working on a fix that should be in the 0.28.3 release in the next couple of days.

[tomasznguyen]

Hi @mlsmaycon. I updated yesterday to version 0.28.3 and I can confirm that the issue is now resolved. Thank you!

[ashish9433]

I am not using selfhosted version but the cloud version. My peers are on 0.28.9, but i see exactly same issue with a caveat that my local DNS is not getting resolved. i.e. After installing the agent i am able to ping google but i am unable to ping my local custom DNS (for example - lab.test.com). Any clues how to fix this?

[Spiritreader]

I am not using selfhosted version but the cloud version. My peers are on 0.28.9, but i see exactly same issue with a caveat that my local DNS is not getting resolved. i.e. After installing the agent i am able to ping google but i am unable to ping my local custom DNS (for example - lab.test.com). Any clues how to fix this?

Is the DNS that resolves lab.test.com for you also running on a netbird peer and you are trying to reach it via netbird?

From what I gather, the following scenario currently doesn't work:

Let's define some example peers:

NetbirdPeerA with example ip 10.10.10.1
NetbirdPeerB with example ip 10.10.10.2 <- runs DNS Server

On NetbirdPeerB you can access the locally running DNS server: nslookup test.lab.com 127.0.0.1 works.

On NetbirdPeerA you cannot access the DNS server running on NetbirdPeerB OVER netbird, as in

$NetbirdPeerA: nslookup test.lab.com 10.10.10.2

will fail.

Similarly, running the same command on NetbirdPeerB with netbird's' peer IP for NetbirdPeerB, as in

$NetbirdPeerB: nslookup test.lab.com 10.10.10.2

will fail.

The DNS request is seemingly auto-rerouted and lost. I've looked into it with TCPdump, and it definitely arrives, but then is routed to.. somewhere? (but not where I expect it to be).

The workaround currently that I see is to not have a DNS running on any netbird peer that you need to reach in your mesh. Install it on a VM or a target machine that is not a peer. Then you can use network routes and either netbird's masquerade feature, or manually via iptables/nftables routing to make it available within your netbird mesh.