netbirdio / netbird

Connect your devices into a secure WireGuard®-based overlay network with SSO, MFA and granular access controls.
https://netbird.io
BSD 3-Clause "New" or "Revised" License
10.5k stars 472 forks source link

NetworkManager routes are deleted if Internal nameserver is configured #1559

Open eriverosr opened 6 months ago

eriverosr commented 6 months ago

Describe the problem

Since 0.25.6, if there is a NetBird peer in Linux with Network Manager and an internal nameserver configured (using internal-routed IPs), the routes assigned to the peer disappear after initial connection.

To Reproduce

Steps to reproduce the behavior:

  1. Authorize two peers with a self-hosted NetBird instance. One of them should be using Linux with NetworkManager and the other one should be in a network with an internal nameserver.
  2. Add the internal nameserver not NetBird and assign it to the Linux peer.
  3. Add a custom route to the nameserver (through the machine in the same internal network) and assign it to the Linux peer.
  4. Monitor the network routes in the Linux peer with watch -n 0.5 ip route
  5. Try to connect both peers.
  6. Check the watch command. The routes should appear temporally, but then they disappear.

Expected behavior

Routes shouldn't be deleted if the peer has assigned an internal nameserver.

The problem does not occur if no nameserver or a public nameserver (Google/Cloudflare/Quad9/etc) is assigned to the Linux host.

The problem does not occur at all with version 0.25.5.

The problem does not occur with the Windows or macOS Client (I just have tested it with Arch Linux. I haven't got the opportunity to test it with Ubuntu)

Are you using NetBird Cloud?

No. I'm using the self-hosted version.

NetBird version

Client: 0.25.6, installed in an Arch Linux machine with NetworkManager (version 0.25.5 doesn't have this problem).

Screenshots

Nameserver:

imagen

Relevant Routes:

image 1

ACLs:

image 2

Peer:

image 3

Additional context

Logs during the client connection with NetBird Client v0.25.5 (everything works)


2024-02-10T19:20:31-03:00 DEBG client/internal/login.go:93: connecting to the Management service https://HOST:PORT
2024-02-10T19:20:32-03:00 DEBG client/internal/login.go:63: connected to the Management service https://HOST:PORT
2024-02-10T19:20:32-03:00 INFO client/internal/connect.go:94: starting NetBird client version development
2024-02-10T19:20:32-03:00 DEBG client/internal/connect.go:149: connecting to the Management service HOST:PORT
2024-02-10T19:20:33-03:00 DEBG client/internal/connect.go:157: connected to the Management service HOST:PORT
2024-02-10T19:20:34-03:00 DEBG signal/client/grpc.go:91: connected to Signal Service: HOST:SIGNALPORT
2024-02-10T19:20:34-03:00 DEBG client/internal/wgproxy/proxy_ebpf.go:36: instantiate ebpf proxy
2024-02-10T19:20:34-03:00 DEBG client/internal/ebpf/ebpf/wg_proxy_linux.go:11: load ebpf WG proxy
2024-02-10T19:20:34-03:00 INFO client/internal/wgproxy/proxy_ebpf.go:79: local wg proxy listening on: 3128
2024-02-10T19:20:34-03:00 DEBG iface/tun_kernel_linux.go:70: adding device: wt0
2024-02-10T19:20:34-03:00 DEBG iface/tun_kernel_linux.go:198: adding address 100.89.144.116/16 to interface: wt0
2024-02-10T19:20:34-03:00 DEBG iface/tun_kernel_linux.go:86: setting MTU: 1280 interface: wt0
2024-02-10T19:20:34-03:00 DEBG iface/wg_configurer_kernel.go:27: adding Wireguard private key
2024-02-10T19:20:34-03:00 DEBG client/firewall/create_linux.go:51: creating an nftables firewall manager
2024-02-10T19:20:34-03:00 WARN client/firewall/nftables/route_linux.go:69: table 'filter' not found for forward rules
2024-02-10T19:20:34-03:00 DEBG iface/tun_kernel_linux.go:110: bringing up interface: wt0
2024-02-10T19:20:34-03:00 DEBG iface/tun_kernel_linux.go:130: device is ready to use: wt0
2024-02-10T19:20:34-03:00 DEBG client/internal/dns/host_linux.go:47: discovered mode is: file
2024-02-10T19:20:34-03:00 DEBG signal/client/grpc.go:136: signal connection state READY
2024-02-10T19:20:34-03:00 INFO signal/client/grpc.go:157: connected to the Signal Service stream
2024-02-10T19:20:34-03:00 DEBG client/internal/engine.go:568: connecting to Management Service updates stream
2024-02-10T19:20:34-03:00 INFO client/internal/connect.go:231: Netbird engine started, my IP is: 100.89.144.116/16
2024-02-10T19:20:34-03:00 DEBG management/client/grpc.go:116: management connection state READY
2024-02-10T19:20:34-03:00 INFO management/client/grpc.go:143: connected to the Management Service stream
2024-02-10T19:20:34-03:00 DEBG management/client/grpc.go:249: got an update message from Management Service
2024-02-10T19:20:34-03:00 DEBG client/internal/engine.go:594: got TURNs update from Management Service, updating
2024-02-10T19:20:34-03:00 DEBG client/internal/engine.go:576: got STUNs update from Management Service, updating
2024-02-10T19:20:34-03:00 DEBG client/internal/engine.go:625: got peers update from Management Service, total peers to connect to = 3
2024-02-10T19:20:34-03:00 DEBG client/internal/engine.go:761: added offline peer PEER
...
2024-02-10T19:20:34-03:00 DEBG client/internal/engine.go:850: creating peer connection PEER1
2024-02-10T19:20:34-03:00 DEBG client/internal/engine.go:850: creating peer connection PEER2
2024-02-10T19:20:34-03:00 DEBG client/internal/engine.go:850: creating peer connection PEER3
2024-02-10T19:20:34-03:00 DEBG client/internal/routemanager/client.go:284: received a new client network route update for 172.16.118.0/24
2024-02-10T19:20:34-03:00 WARN client/internal/routemanager/client.go:121: the network 172.16.118.0/24 has not been assigned a routing peer as no peers from the list [PEER1] are currently connected
2024-02-10T19:20:34-03:00 DEBG client/internal/dns/service_listener.go:78: starting dns on 100.89.144.116:53
2024-02-10T19:20:34-03:00 DEBG client/internal/dns/file_linux.go:92: creating managed file /etc/resolv.conf
2024-02-10T19:20:34-03:00 INFO client/internal/dns/file_linux.go:102: created a NetBird managed /etc/resolv.conf file with your DNS settings. Added 1 search domains. Search list: [netbird.HOST]
2024-02-10T19:20:34-03:00 INFO client/internal/dns/file_repair_linux.go:47: start to watch resolv.conf
2024-02-10T19:20:36-03:00 WARN client/internal/dns/upstream.go:169: probing upstream nameserver 172.16.118.92:53: read udp 192.168.3.132:57488->172.16.118.92:53: i/o timeout
2024-02-10T19:20:36-03:00 WARN client/internal/dns/upstream.go:249: upstream resolving is Disabled for 30s
2024-02-10T19:20:36-03:00 INFO [nameservers: [{172.16.118.92 udp 53}]] client/internal/dns/server.go:467: temporary deactivate nameservers group due timeout
2024-02-10T19:20:36-03:00 DEBG client/internal/dns/file_repair_linux.go:71: check resolv.conf parameters: search domains: [], name servers: [1.1.1.1 1.0.0.1 fe80::3266:d0ff:fed9:ddb4%wlp2s0], others: []
2024-02-10T19:20:36-03:00 INFO client/internal/dns/file_repair_linux.go:76: broken params in resolv.conf, repairing it...
2024-02-10T19:20:36-03:00 DEBG client/internal/dns/file_linux.go:92: creating managed file /etc/resolv.conf
2024-02-10T19:20:36-03:00 ERRO [nameservers: [{172.16.118.92 udp 53}], error: unable to configure DNS for this peer using file manager without a nameserver group with all domains configured] client/internal/dns/server.go:486: fail to apply nameserver deactivation on the host
2024-02-10T19:20:36-03:00 INFO client/internal/acl/manager.go:52: ACL rules processed in: 816.021µs, total rules count: 2
2024-02-10T19:20:36-03:00 DEBG client/internal/peer/conn.go:254: trying to connect to peer PEER2
2024-02-10T19:20:36-03:00 DEBG client/internal/peer/conn.go:254: trying to connect to peer PEER3
2024-02-10T19:20:36-03:00 DEBG client/internal/peer/conn.go:254: trying to connect to peer PEER1
2024-02-10T19:20:36-03:00 INFO client/internal/dns/file_linux.go:102: created a NetBird managed /etc/resolv.conf file with your DNS settings. Added 1 search domains. Search list: [netbird.HOST]
2024-02-10T19:20:36-03:00 DEBG client/internal/dns/file_repair_linux.go:71: check resolv.conf parameters: search domains: [netbird.HOST], name servers: [100.89.144.116 1.1.1.1 1.0.0.1 fe80::3266:d0ff:fed9:ddb4%wlp2s0], others: []
2024-02-10T19:20:36-03:00 DEBG client/internal/peer/conn.go:285: connection offer sent to peer PEER2, waiting for the confirmation
2024-02-10T19:20:36-03:00 DEBG client/internal/peer/conn.go:285: connection offer sent to peer PEER3, waiting for the confirmation
2024-02-10T19:20:36-03:00 DEBG client/internal/peer/conn.go:285: connection offer sent to peer PEER1, waiting for the confirmation
2024-02-10T19:20:37-03:00 WARN [error: read udp 192.168.3.132:33732->172.16.118.92:53: i/o timeout, upstream: 172.16.118.92:53] client/internal/dns/upstream.go:89: got an error while connecting to upstream
2024-02-10T19:20:37-03:00 ERRO client/internal/dns/upstream.go:121: all queries to the upstream nameservers failed with timeout
2024-02-10T19:20:37-03:00 WARN [error: read udp 192.168.3.132:57474->172.16.118.92:53: i/o timeout, upstream: 172.16.118.92:53] client/internal/dns/upstream.go:89: got an error while connecting to upstream
2024-02-10T19:20:37-03:00 ERRO client/internal/dns/upstream.go:121: all queries to the upstream nameservers failed with timeout
2024-02-10T19:20:37-03:00 WARN [upstream: 172.16.118.92:53, error: read udp 192.168.3.132:58038->172.16.118.92:53: i/o timeout] client/internal/dns/upstream.go:89: got an error while connecting to upstream
2024-02-10T19:20:37-03:00 ERRO client/internal/dns/upstream.go:121: all queries to the upstream nameservers failed with timeout
2024-02-10T19:20:37-03:00 WARN [error: read udp 192.168.3.132:55034->172.16.118.92:53: i/o timeout, upstream: 172.16.118.92:53] client/internal/dns/upstream.go:89: got an error while connecting to upstream
2024-02-10T19:20:37-03:00 ERRO client/internal/dns/upstream.go:121: all queries to the upstream nameservers failed with timeout
2024-02-10T19:20:37-03:00 WARN [error: read udp 192.168.3.132:57141->172.16.118.92:53: i/o timeout, upstream: 172.16.118.92:53] client/internal/dns/upstream.go:89: got an error while connecting to upstream
2024-02-10T19:20:37-03:00 ERRO client/internal/dns/upstream.go:121: all queries to the upstream nameservers failed with timeout
2024-02-10T19:20:37-03:00 DEBG client/internal/peer/conn.go:673: OnRemoteAnswer from peer PEER3 on status Disconnected
2024-02-10T19:20:37-03:00 DEBG client/internal/peer/conn.go:673: OnRemoteAnswer from peer PEER1 on status Disconnected
2024-02-10T19:20:37-03:00 DEBG client/internal/peer/conn.go:306: received connection confirmation from peer PEER1 running version 0.25.2 and with remote WireGuard listen port 51820
2024-02-10T19:20:37-03:00 DEBG client/internal/peer/conn.go:306: received connection confirmation from peer PEER3 running version 0.25.7 and with remote WireGuard listen port 51820
2024-02-10T19:20:37-03:00 DEBG client/internal/peer/conn.go:571: peer PEER3 ICE ConnectionState has changed to Checking
2024-02-10T19:20:37-03:00 DEBG client/internal/peer/conn.go:571: peer PEER1 ICE ConnectionState has changed to Checking
2024-02-10T19:20:37-03:00 DEBG client/internal/peer/conn.go:554: discovered local candidate udp4 host 192.168.3.132:51820
2024-02-10T19:20:37-03:00 DEBG client/internal/peer/conn.go:554: discovered local candidate udp4 host 192.168.3.132:51820
2024-02-10T19:20:37-03:00 DEBG client/internal/peer/conn.go:687: OnRemoteCandidate from peer PEER3 -> udp4 host 172.16.118.23:51820
2024-02-10T19:20:37-03:00 DEBG client/internal/peer/conn.go:687: OnRemoteCandidate from peer PEER1 -> udp4 host 172.16.118.92:51820
2024-02-10T19:20:37-03:00 DEBG iface/bind/udp_mux.go:346: ICE: registered 172.16.118.23:51820 for ID1
2024-02-10T19:20:37-03:00 DEBG iface/bind/udp_mux.go:346: ICE: registered 172.16.118.92:51820 for ID2
2024-02-10T19:20:37-03:00 DEBG client/internal/peer/conn.go:687: OnRemoteCandidate from peer PEER3 -> udp4 srflx PEER3_IP:51820 related 0.0.0.0:51820
2024-02-10T19:20:37-03:00 DEBG client/internal/peer/conn.go:687: OnRemoteCandidate from peer PEER1 -> udp4 srflx PEER1_IP:51820 related 0.0.0.0:51820
2024-02-10T19:20:37-03:00 DEBG iface/bind/udp_mux.go:346: ICE: registered PEER1_IP:51820 for ID2
2024-02-10T19:20:37-03:00 DEBG iface/bind/udp_mux.go:346: ICE: registered PEER3_IP:51820 for ID1
2024-02-10T19:20:37-03:00 DEBG client/internal/peer/conn.go:687: OnRemoteCandidate from peer PEER1 -> udp4 relay RELAY_IP:56907 related 0.0.0.0:57644
2024-02-10T19:20:37-03:00 DEBG iface/bind/udp_mux.go:346: ICE: registered RELAY_IP:56907 for ID2
2024-02-10T19:20:37-03:00 DEBG client/internal/peer/conn.go:687: OnRemoteCandidate from peer PEER3 -> udp4 relay RELAY_IP:54247 related 0.0.0.0:59694
2024-02-10T19:20:37-03:00 DEBG iface/bind/udp_mux.go:346: ICE: registered RELAY_IP:54247 for ID1
2024-02-10T19:20:37-03:00 DEBG client/internal/peer/conn.go:565: selected candidate pair [local <-> remote] -> [udp4 host 192.168.3.132:51820 <-> udp4 srflx PEER3_IP:51820 related 0.0.0.0:51820], peer PEER3
2024-02-10T19:20:37-03:00 DEBG client/internal/peer/conn.go:565: selected candidate pair [local <-> remote] -> [udp4 host 192.168.3.132:51820 <-> udp4 srflx PEER1_IP:51820 related 0.0.0.0:51820], peer PEER1
2024-02-10T19:20:37-03:00 DEBG client/internal/peer/conn.go:571: peer PEER1 ICE ConnectionState has changed to Connected
2024-02-10T19:20:37-03:00 DEBG client/internal/peer/conn.go:571: peer PEER3 ICE ConnectionState has changed to Connected
2024-02-10T19:20:37-03:00 DEBG iface/iface.go:79: updating interface wt0 peer PEER1, endpoint PEER1_IP:51820
2024-02-10T19:20:37-03:00 INFO client/internal/peer/conn.go:357: connected to peer PEER1, endpoint address: PEER1_IP:51820
2024-02-10T19:20:37-03:00 DEBG iface/iface.go:79: updating interface wt0 peer PEER3, endpoint PEER3_IP:51820
2024-02-10T19:20:37-03:00 INFO client/internal/routemanager/client.go:124: new chosen route is ROUTE1 with peer PEER1 with score 2 for network 172.16.118.0/24
2024-02-10T19:20:37-03:00 DEBG client/internal/routemanager/client.go:143: triggered route state update for Peer PEER1, state: Connected
2024-02-10T19:20:37-03:00 INFO client/internal/peer/conn.go:357: connected to peer PEER3, endpoint address: PEER3_IP:51820
2024-02-10T19:20:37-03:00 DEBG iface/iface.go:97: adding allowed IP to interface wt0 and peer PEER1: allowed IP 172.16.118.0/24 
2024-02-10T19:20:39-03:00 WARN [error: read udp 192.168.3.132:38824->172.16.118.92:53: i/o timeout, upstream: 172.16.118.92:53] client/internal/dns/upstream.go:89: got an error while connecting to upstream
2024-02-10T19:20:39-03:00 ERRO client/internal/dns/upstream.go:121: all queries to the upstream nameservers failed with timeout
2024-02-10T19:20:39-03:00 WARN [error: read udp 192.168.3.132:46233->172.16.118.92:53: i/o timeout, upstream: 172.16.118.92:53] client/internal/dns/upstream.go:89: got an error while connecting to upstream
2024-02-10T19:20:39-03:00 ERRO client/internal/dns/upstream.go:121: all queries to the upstream nameservers failed with timeout
2024-02-10T19:20:41-03:00 WARN [error: read udp 100.89.144.116:52847->172.16.118.92:53: i/o timeout, upstream: 172.16.118.92:53] client/internal/dns/upstream.go:89: got an error while connecting to upstream
2024-02-10T19:20:41-03:00 ERRO client/internal/dns/upstream.go:121: all queries to the upstream nameservers failed with timeout
2024-02-10T19:20:41-03:00 WARN [error: read udp 100.89.144.116:47439->172.16.118.92:53: i/o timeout, upstream: 172.16.118.92:53] client/internal/dns/upstream.go:89: got an error while connecting to upstream
2024-02-10T19:20:41-03:00 ERRO client/internal/dns/upstream.go:121: all queries to the upstream nameservers failed with timeout
2024-02-10T19:20:42-03:00 WARN [error: read udp 100.89.144.116:50637->172.16.118.92:53: i/o timeout, upstream: 172.16.118.92:53] client/internal/dns/upstream.go:89: got an error while connecting to upstream
2024-02-10T19:20:42-03:00 ERRO client/internal/dns/upstream.go:121: all queries to the upstream nameservers failed with timeout
2024-02-10T19:20:42-03:00 WARN [error: read udp 100.89.144.116:49203->172.16.118.92:53: i/o timeout, upstream: 172.16.118.92:53] client/internal/dns/upstream.go:89: got an error while connecting to upstream
2024-02-10T19:20:42-03:00 ERRO client/internal/dns/upstream.go:121: all queries to the upstream nameservers failed with timeout
2024-02-10T19:20:42-03:00 WARN [error: read udp 100.89.144.116:54192->172.16.118.92:53: i/o timeout, upstream: 172.16.118.92:53] client/internal/dns/upstream.go:89: got an error while connecting to upstream
2024-02-10T19:20:42-03:00 ERRO client/internal/dns/upstream.go:121: all queries to the upstream nameservers failed with timeout
2024-02-10T19:20:42-03:00 WARN [error: read udp 100.89.144.116:49999->172.16.118.92:53: i/o timeout, upstream: 172.16.118.92:53] client/internal/dns/upstream.go:89: got an error while connecting to upstream
2024-02-10T19:20:42-03:00 ERRO client/internal/dns/upstream.go:121: all queries to the upstream nameservers failed with timeout
2024-02-10T19:20:42-03:00 DEBG client/internal/peer/conn.go:554: discovered local candidate udp4 srflx PUBLIC_IP:51820 related 0.0.0.0:51820
2024-02-10T19:20:42-03:00 DEBG client/internal/peer/conn.go:554: discovered local candidate udp4 srflx PUBLIC_IP:51820 related 0.0.0.0:51820
2024-02-10T19:20:42-03:00 DEBG client/internal/peer/conn.go:554: discovered local candidate udp4 relay RELAY_IP:63550 related 0.0.0.0:45853
2024-02-10T19:20:42-03:00 DEBG client/internal/peer/conn.go:554: discovered local candidate udp4 relay RELAY_IP:57298 related 0.0.0.0:41368
2024-02-10T19:20:42-03:00 INFO client/internal/dns/upstream.go:225: upstreams [172.16.118.92:53] are responsive again. Adding them back to system
2024-02-10T19:20:42-03:00 DEBG [nameservers: [{172.16.118.92 udp 53}]] client/internal/dns/server.go:502: reactivate temporary disabled nameserver group
2024-02-10T19:20:42-03:00 DEBG client/internal/dns/file_linux.go:92: creating managed file /etc/resolv.conf
2024-02-10T19:20:42-03:00 INFO client/internal/dns/file_linux.go:102: created a NetBird managed /etc/resolv.conf file with your DNS settings. Added 1 search domains. Search list: [netbird.HOST]
2024-02-10T19:20:42-03:00 INFO client/internal/dns/file_repair_linux.go:47: start to watch resolv.conf

Logs during the client connection with NetBird Client v0.25.6 (routes dissapear)

2024-02-10T19:18:10-03:00 DEBG client/internal/login.go:93: connecting to the Management service https://HOST:PORT
2024-02-10T19:18:11-03:00 DEBG client/internal/login.go:63: connected to the Management service https://HOST:PORT
2024-02-10T19:18:11-03:00 INFO client/internal/connect.go:95: starting NetBird client version development
2024-02-10T19:18:11-03:00 DEBG client/internal/connect.go:156: connecting to the Management service HOST:PORT
2024-02-10T19:18:12-03:00 DEBG client/internal/connect.go:164: connected to the Management service HOST:PORT
2024-02-10T19:18:13-03:00 DEBG signal/client/grpc.go:91: connected to Signal Service: HOST:SIGNALPORT
2024-02-10T19:18:13-03:00 DEBG client/internal/wgproxy/proxy_ebpf.go:36: instantiate ebpf proxy
2024-02-10T19:18:13-03:00 DEBG client/internal/ebpf/ebpf/wg_proxy_linux.go:11: load ebpf WG proxy
2024-02-10T19:18:13-03:00 INFO client/internal/wgproxy/proxy_ebpf.go:79: local wg proxy listening on: 3128
2024-02-10T19:18:13-03:00 DEBG iface/tun_kernel_linux.go:70: adding device: wt0
2024-02-10T19:18:13-03:00 DEBG iface/tun_kernel_linux.go:198: adding address 100.89.144.116/16 to interface: wt0
2024-02-10T19:18:13-03:00 DEBG iface/tun_kernel_linux.go:86: setting MTU: 1280 interface: wt0
2024-02-10T19:18:13-03:00 DEBG iface/wg_configurer_kernel.go:27: adding Wireguard private key
2024-02-10T19:18:13-03:00 DEBG client/firewall/create_linux.go:51: creating an nftables firewall manager
2024-02-10T19:18:13-03:00 WARN client/firewall/nftables/route_linux.go:69: table 'filter' not found for forward rules
2024-02-10T19:18:13-03:00 DEBG iface/tun_kernel_linux.go:110: bringing up interface: wt0
2024-02-10T19:18:13-03:00 DEBG iface/tun_kernel_linux.go:130: device is ready to use: wt0
2024-02-10T19:18:13-03:00 DEBG client/internal/dns/network_manager_linux.go:332: network manager constraints [>= 1.16, < 1.27 | >= 1.44, < 1.45] met: true
2024-02-10T19:18:13-03:00 DEBG client/internal/dns/host_linux.go:68: discovered mode is: networkManager
2024-02-10T19:18:13-03:00 DEBG client/internal/dns/network_manager_linux.go:92: got network manager dbus Link Object: /org/freedesktop/NetworkManager/Devices/57 from net interface wt0
2024-02-10T19:18:13-03:00 DEBG signal/client/grpc.go:136: signal connection state READY
2024-02-10T19:18:13-03:00 INFO signal/client/grpc.go:157: connected to the Signal Service stream
2024-02-10T19:18:13-03:00 DEBG client/internal/engine.go:568: connecting to Management Service updates stream
2024-02-10T19:18:13-03:00 INFO client/internal/connect.go:238: Netbird engine started, my IP is: 100.89.144.116/16
2024-02-10T19:18:13-03:00 DEBG management/client/grpc.go:116: management connection state READY
2024-02-10T19:18:13-03:00 INFO management/client/grpc.go:143: connected to the Management Service stream
2024-02-10T19:18:13-03:00 DEBG management/client/grpc.go:249: got an update message from Management Service
2024-02-10T19:18:13-03:00 DEBG client/internal/engine.go:594: got TURNs update from Management Service, updating
2024-02-10T19:18:13-03:00 DEBG client/internal/engine.go:576: got STUNs update from Management Service, updating
2024-02-10T19:18:13-03:00 DEBG client/internal/engine.go:625: got peers update from Management Service, total peers to connect to = 3
2024-02-10T19:18:13-03:00 DEBG client/internal/engine.go:761: added offline peer peers
...
2024-02-10T19:18:13-03:00 DEBG client/internal/engine.go:850: creating peer connection PEER1
2024-02-10T19:18:13-03:00 DEBG client/internal/engine.go:850: creating peer connection PEER2
2024-02-10T19:18:13-03:00 DEBG client/internal/engine.go:850: creating peer connection PEER3
2024-02-10T19:18:13-03:00 DEBG client/internal/routemanager/client.go:284: received a new client network route update for 172.16.118.0/24
2024-02-10T19:18:13-03:00 WARN client/internal/routemanager/client.go:121: the network 172.16.118.0/24 has not been assigned a routing peer as no peers from the list [PEER1] are currently connected
2024-02-10T19:18:13-03:00 DEBG client/internal/dns/service_listener.go:69: starting dns on 100.89.144.116:53
2024-02-10T19:18:13-03:00 INFO client/internal/dns/network_manager_linux.go:140: configured 100.89.144.116:53 as main DNS forwarder for this peer
2024-02-10T19:18:13-03:00 INFO client/internal/dns/network_manager_linux.go:160: adding 1 search domains and 0 match domains. Search list: [netbird.HOST.] , Match list: []
2024-02-10T19:18:15-03:00 WARN client/internal/dns/upstream.go:171: probing upstream nameserver 172.16.118.92:53: read udp 192.168.3.132:58095->172.16.118.92:53: i/o timeout
2024-02-10T19:18:15-03:00 WARN client/internal/dns/upstream.go:251: upstream resolving is Disabled for 30s
2024-02-10T19:18:15-03:00 INFO [nameservers: [{172.16.118.92 udp 53}]] client/internal/dns/server.go:470: temporary deactivate nameservers group due timeout
2024-02-10T19:18:15-03:00 INFO client/internal/dns/network_manager_linux.go:160: adding 1 search domains and 0 match domains. Search list: [netbird.HOST.] , Match list: []
2024-02-10T19:18:15-03:00 INFO client/internal/acl/manager.go:52: ACL rules processed in: 622.493µs, total rules count: 2
2024-02-10T19:18:15-03:00 DEBG client/internal/peer/conn.go:254: trying to connect to peer PEER3
2024-02-10T19:18:15-03:00 DEBG client/internal/peer/conn.go:254: trying to connect to peer PEER2
2024-02-10T19:18:15-03:00 DEBG client/internal/peer/conn.go:254: trying to connect to peer PEER1
2024-02-10T19:18:15-03:00 DEBG client/internal/peer/conn.go:285: connection offer sent to peer PEER3, waiting for the confirmation
2024-02-10T19:18:15-03:00 DEBG client/internal/peer/conn.go:285: connection offer sent to peer PEER2, waiting for the confirmation
2024-02-10T19:18:15-03:00 DEBG client/internal/peer/conn.go:285: connection offer sent to peer PEER1, waiting for the confirmation
2024-02-10T19:18:15-03:00 DEBG client/internal/peer/conn.go:673: OnRemoteAnswer from peer PEER3 on status Disconnected
2024-02-10T19:18:15-03:00 DEBG client/internal/peer/conn.go:306: received connection confirmation from peer PEER3 running version 0.25.7 and with remote WireGuard listen port 51820
2024-02-10T19:18:15-03:00 DEBG client/internal/peer/conn.go:571: peer PEER3 ICE ConnectionState has changed to Checking
2024-02-10T19:18:15-03:00 DEBG client/internal/peer/conn.go:673: OnRemoteAnswer from peer PEER1 on status Disconnected
2024-02-10T19:18:15-03:00 DEBG client/internal/peer/conn.go:306: received connection confirmation from peer PEER1 running version 0.25.2 and with remote WireGuard listen port 51820
2024-02-10T19:18:15-03:00 DEBG client/internal/peer/conn.go:571: peer PEER1 ICE ConnectionState has changed to Checking
2024-02-10T19:18:15-03:00 DEBG client/internal/peer/conn.go:554: discovered local candidate udp4 host 192.168.3.132:51820
2024-02-10T19:18:15-03:00 DEBG client/internal/peer/conn.go:554: discovered local candidate udp4 host 192.168.3.132:51820
2024-02-10T19:18:16-03:00 WARN [error: read udp 192.168.3.132:37328->172.16.118.92:53: i/o timeout, upstream: 172.16.118.92:53] client/internal/dns/upstream.go:91: got an error while connecting to upstream
2024-02-10T19:18:16-03:00 ERRO client/internal/dns/upstream.go:123: all queries to the upstream nameservers failed with timeout
2024-02-10T19:18:16-03:00 WARN [error: read udp 192.168.3.132:33131->172.16.118.92:53: i/o timeout, upstream: 172.16.118.92:53] client/internal/dns/upstream.go:91: got an error while connecting to upstream
2024-02-10T19:18:16-03:00 ERRO client/internal/dns/upstream.go:123: all queries to the upstream nameservers failed with timeout
2024-02-10T19:18:16-03:00 WARN [error: read udp 192.168.3.132:37299->172.16.118.92:53: i/o timeout, upstream: 172.16.118.92:53] client/internal/dns/upstream.go:91: got an error while connecting to upstream
2024-02-10T19:18:16-03:00 ERRO client/internal/dns/upstream.go:123: all queries to the upstream nameservers failed with timeout
2024-02-10T19:18:16-03:00 WARN [error: read udp 192.168.3.132:49865->172.16.118.92:53: i/o timeout, upstream: 172.16.118.92:53] client/internal/dns/upstream.go:91: got an error while connecting to upstream
2024-02-10T19:18:16-03:00 ERRO client/internal/dns/upstream.go:123: all queries to the upstream nameservers failed with timeout
2024-02-10T19:18:16-03:00 WARN [error: read udp 192.168.3.132:46213->172.16.118.92:53: i/o timeout, upstream: 172.16.118.92:53] client/internal/dns/upstream.go:91: got an error while connecting to upstream
2024-02-10T19:18:16-03:00 ERRO client/internal/dns/upstream.go:123: all queries to the upstream nameservers failed with timeout
2024-02-10T19:18:16-03:00 WARN [upstream: 172.16.118.92:53, error: read udp 192.168.3.132:45746->172.16.118.92:53: i/o timeout] client/internal/dns/upstream.go:91: got an error while connecting to upstream
2024-02-10T19:18:16-03:00 ERRO client/internal/dns/upstream.go:123: all queries to the upstream nameservers failed with timeout
2024-02-10T19:18:16-03:00 WARN [error: read udp 192.168.3.132:46258->172.16.118.92:53: i/o timeout, upstream: 172.16.118.92:53] client/internal/dns/upstream.go:91: got an error while connecting to upstream
2024-02-10T19:18:16-03:00 WARN [error: read udp 192.168.3.132:45996->172.16.118.92:53: i/o timeout, upstream: 172.16.118.92:53] client/internal/dns/upstream.go:91: got an error while connecting to upstream
2024-02-10T19:18:16-03:00 ERRO client/internal/dns/upstream.go:123: all queries to the upstream nameservers failed with timeout
2024-02-10T19:18:16-03:00 ERRO client/internal/dns/upstream.go:123: all queries to the upstream nameservers failed with timeout
2024-02-10T19:18:16-03:00 DEBG client/internal/peer/conn.go:687: OnRemoteCandidate from peer PEER3 -> udp4 host 172.16.118.23:51820
2024-02-10T19:18:16-03:00 DEBG client/internal/peer/conn.go:687: OnRemoteCandidate from peer PEER1 -> udp4 host 172.16.118.92:51820
2024-02-10T19:18:16-03:00 DEBG iface/bind/udp_mux.go:346: ICE: registered 172.16.118.23:51820 for ID1
2024-02-10T19:18:16-03:00 DEBG iface/bind/udp_mux.go:346: ICE: registered 172.16.118.92:51820 for ID2
2024-02-10T19:18:16-03:00 DEBG client/internal/peer/conn.go:687: OnRemoteCandidate from peer PEER1 -> udp4 srflx PEER3_IP:51820 related 0.0.0.0:51820
2024-02-10T19:18:16-03:00 DEBG client/internal/peer/conn.go:687: OnRemoteCandidate from peer PEER3 -> udp4 srflx PEER1_IP:51820 related 0.0.0.0:51820
2024-02-10T19:18:16-03:00 DEBG iface/bind/udp_mux.go:346: ICE: registered PEER3_IP:51820 for ID2
2024-02-10T19:18:16-03:00 DEBG iface/bind/udp_mux.go:346: ICE: registered PEER1_IP:51820 for ID1
2024-02-10T19:18:17-03:00 DEBG client/internal/peer/conn.go:687: OnRemoteCandidate from peer PEER1 -> udp4 relay RELAY_IP:64869 related 0.0.0.0:35826
2024-02-10T19:18:17-03:00 DEBG iface/bind/udp_mux.go:346: ICE: registered RELAY_IP:64869 for ID2
2024-02-10T19:18:17-03:00 DEBG client/internal/peer/conn.go:687: OnRemoteCandidate from peer PEER3 -> udp4 relay RELAY_IP:63737 related 0.0.0.0:46912
2024-02-10T19:18:17-03:00 DEBG iface/bind/udp_mux.go:346: ICE: registered RELAY_IP:63737 for ID1
2024-02-10T19:18:17-03:00 WARN [error: read udp 192.168.3.132:46630->172.16.118.92:53: i/o timeout, upstream: 172.16.118.92:53] client/internal/dns/upstream.go:91: got an error while connecting to upstream
2024-02-10T19:18:17-03:00 ERRO client/internal/dns/upstream.go:123: all queries to the upstream nameservers failed with timeout
2024-02-10T19:18:17-03:00 WARN [error: read udp 192.168.3.132:37949->172.16.118.92:53: i/o timeout, upstream: 172.16.118.92:53] client/internal/dns/upstream.go:91: got an error while connecting to upstream
2024-02-10T19:18:17-03:00 ERRO client/internal/dns/upstream.go:123: all queries to the upstream nameservers failed with timeout
2024-02-10T19:18:17-03:00 WARN [upstream: 172.16.118.92:53, error: read udp 192.168.3.132:38371->172.16.118.92:53: i/o timeout] client/internal/dns/upstream.go:91: got an error while connecting to upstream
2024-02-10T19:18:17-03:00 ERRO client/internal/dns/upstream.go:123: all queries to the upstream nameservers failed with timeout
2024-02-10T19:18:17-03:00 WARN [error: read udp 192.168.3.132:51819->172.16.118.92:53: i/o timeout, upstream: 172.16.118.92:53] client/internal/dns/upstream.go:91: got an error while connecting to upstream
2024-02-10T19:18:17-03:00 ERRO client/internal/dns/upstream.go:123: all queries to the upstream nameservers failed with timeout
2024-02-10T19:18:18-03:00 WARN [error: read udp 192.168.3.132:35261->172.16.118.92:53: i/o timeout, upstream: 172.16.118.92:53] client/internal/dns/upstream.go:91: got an error while connecting to upstream
2024-02-10T19:18:18-03:00 ERRO client/internal/dns/upstream.go:123: all queries to the upstream nameservers failed with timeout
2024-02-10T19:18:18-03:00 WARN [error: read udp 192.168.3.132:60709->172.16.118.92:53: i/o timeout, upstream: 172.16.118.92:53] client/internal/dns/upstream.go:91: got an error while connecting to upstream
2024-02-10T19:18:18-03:00 ERRO client/internal/dns/upstream.go:123: all queries to the upstream nameservers failed with timeout
2024-02-10T19:18:21-03:00 WARN [error: read udp 192.168.3.132:45713->172.16.118.92:53: i/o timeout, upstream: 172.16.118.92:53] client/internal/dns/upstream.go:91: got an error while connecting to upstream
2024-02-10T19:18:21-03:00 ERRO client/internal/dns/upstream.go:123: all queries to the upstream nameservers failed with timeout
2024-02-10T19:18:21-03:00 WARN [error: read udp 192.168.3.132:58578->172.16.118.92:53: i/o timeout, upstream: 172.16.118.92:53] client/internal/dns/upstream.go:91: got an error while connecting to upstream
2024-02-10T19:18:21-03:00 ERRO client/internal/dns/upstream.go:123: all queries to the upstream nameservers failed with timeout
2024-02-10T19:18:21-03:00 WARN [error: read udp 192.168.3.132:40189->172.16.118.92:53: i/o timeout, upstream: 172.16.118.92:53] client/internal/dns/upstream.go:91: got an error while connecting to upstream
2024-02-10T19:18:21-03:00 ERRO client/internal/dns/upstream.go:123: all queries to the upstream nameservers failed with timeout
2024-02-10T19:18:21-03:00 WARN [error: read udp 192.168.3.132:40850->172.16.118.92:53: i/o timeout, upstream: 172.16.118.92:53] client/internal/dns/upstream.go:91: got an error while connecting to upstream
2024-02-10T19:18:21-03:00 ERRO client/internal/dns/upstream.go:123: all queries to the upstream nameservers failed with timeout
2024-02-10T19:18:21-03:00 WARN [error: read udp 192.168.3.132:38410->172.16.118.92:53: i/o timeout, upstream: 172.16.118.92:53] client/internal/dns/upstream.go:91: got an error while connecting to upstream
2024-02-10T19:18:21-03:00 ERRO client/internal/dns/upstream.go:123: all queries to the upstream nameservers failed with timeout
2024-02-10T19:18:21-03:00 WARN [error: read udp 192.168.3.132:43553->172.16.118.92:53: i/o timeout, upstream: 172.16.118.92:53] client/internal/dns/upstream.go:91: got an error while connecting to upstream
2024-02-10T19:18:21-03:00 ERRO client/internal/dns/upstream.go:123: all queries to the upstream nameservers failed with timeout
2024-02-10T19:18:21-03:00 WARN [error: read udp 192.168.3.132:48762->172.16.118.92:53: i/o timeout, upstream: 172.16.118.92:53] client/internal/dns/upstream.go:91: got an error while connecting to upstream
2024-02-10T19:18:21-03:00 ERRO client/internal/dns/upstream.go:123: all queries to the upstream nameservers failed with timeout
2024-02-10T19:18:21-03:00 WARN [error: read udp 192.168.3.132:35066->172.16.118.92:53: i/o timeout, upstream: 172.16.118.92:53] client/internal/dns/upstream.go:91: got an error while connecting to upstream
2024-02-10T19:18:21-03:00 ERRO client/internal/dns/upstream.go:123: all queries to the upstream nameservers failed with timeout
2024-02-10T19:18:21-03:00 DEBG iface/bind/udp_mux.go:346: ICE: registered 172.16.118.23:51820 for ID1turn:HOST:3478?transport=udp
2024-02-10T19:18:21-03:00 DEBG iface/bind/udp_mux.go:346: ICE: registered PEER1_IP:51820 for ID1turn:HOST:3478?transport=udp
2024-02-10T19:18:21-03:00 DEBG client/internal/peer/conn.go:554: discovered local candidate udp4 srflx PUBLIC_IP:51820 related 0.0.0.0:51820
2024-02-10T19:18:21-03:00 DEBG iface/bind/udp_mux.go:346: ICE: registered 172.16.118.92:51820 for ID2turn:HOST:3478?transport=udp
2024-02-10T19:18:21-03:00 DEBG client/internal/peer/conn.go:554: discovered local candidate udp4 srflx PUBLIC_IP:51820 related 0.0.0.0:51820
2024-02-10T19:18:21-03:00 DEBG iface/bind/udp_mux.go:346: ICE: registered PEER3_IP:51820 for ID2turn:HOST:3478?transport=udp
2024-02-10T19:18:21-03:00 DEBG iface/bind/udp_mux.go:346: ICE: registered RELAY_IP:63737 for ID1turn:HOST:3478?transport=udp
2024-02-10T19:18:21-03:00 DEBG iface/bind/udp_mux.go:346: ICE: registered RELAY_IP:64869 for ID2turn:HOST:3478?transport=udp
2024-02-10T19:18:21-03:00 DEBG client/internal/peer/conn.go:554: discovered local candidate udp4 relay RELAY_IP:61889 related 0.0.0.0:50237
2024-02-10T19:18:21-03:00 DEBG client/internal/peer/conn.go:554: discovered local candidate udp4 relay RELAY_IP:53571 related 0.0.0.0:59695
2024-02-10T19:18:21-03:00 DEBG iface/iface.go:79: updating interface wt0 peer PEER3, endpoint PEER1_IP:51820
2024-02-10T19:18:21-03:00 DEBG client/internal/peer/conn.go:571: peer PEER3 ICE ConnectionState has changed to Connected
2024-02-10T19:18:21-03:00 DEBG client/internal/peer/conn.go:565: selected candidate pair [local <-> remote] -> [udp4 host 192.168.3.132:51820 <-> udp4 srflx PEER1_IP:51820 related 0.0.0.0:51820], peer PEER3
2024-02-10T19:18:21-03:00 INFO client/internal/peer/conn.go:357: connected to peer PEER3, endpoint address: PEER1_IP:51820
2024-02-10T19:18:21-03:00 DEBG iface/iface.go:79: updating interface wt0 peer PEER1, endpoint PEER3_IP:51820
2024-02-10T19:18:21-03:00 DEBG client/internal/peer/conn.go:565: selected candidate pair [local <-> remote] -> [udp4 srflx PUBLIC_IP:51820 related 0.0.0.0:51820 <-> udp4 srflx PEER3_IP:51820 related 0.0.0.0:51820], peer PEER1
2024-02-10T19:18:21-03:00 DEBG client/internal/peer/conn.go:571: peer PEER1 ICE ConnectionState has changed to Connected
2024-02-10T19:18:21-03:00 INFO client/internal/peer/conn.go:357: connected to peer PEER1, endpoint address: PEER3_IP:51820
2024-02-10T19:18:21-03:00 DEBG client/internal/routemanager/client.go:143: triggered route state update for Peer PEER1, state: Connected
2024-02-10T19:18:21-03:00 INFO client/internal/routemanager/client.go:124: new chosen route is cn3v9oqfrlvs739rgos0 with peer PEER1 with score 2 for network 172.16.118.0/24
2024-02-10T19:18:21-03:00 DEBG iface/iface.go:97: adding allowed IP to interface wt0 and peer PEER1: allowed IP 172.16.118.0/24 
2024-02-10T19:18:22-03:00 WARN [error: read udp 192.168.3.132:33239->172.16.118.92:53: i/o timeout, upstream: 172.16.118.92:53] client/internal/dns/upstream.go:91: got an error while connecting to upstream
2024-02-10T19:18:22-03:00 ERRO client/internal/dns/upstream.go:123: all queries to the upstream nameservers failed with timeout
2024-02-10T19:18:22-03:00 WARN [upstream: 172.16.118.92:53, error: read udp 192.168.3.132:34859->172.16.118.92:53: i/o timeout] client/internal/dns/upstream.go:91: got an error while connecting to upstream
2024-02-10T19:18:22-03:00 ERRO client/internal/dns/upstream.go:123: all queries to the upstream nameservers failed with timeout
2024-02-10T19:18:22-03:00 WARN [upstream: 172.16.118.92:53, error: read udp 192.168.3.132:50550->172.16.118.92:53: i/o timeout] client/internal/dns/upstream.go:91: got an error while connecting to upstream
2024-02-10T19:18:22-03:00 ERRO client/internal/dns/upstream.go:123: all queries to the upstream nameservers failed with timeout
2024-02-10T19:18:23-03:00 WARN [error: read udp 192.168.3.132:48523->172.16.118.92:53: i/o timeout, upstream: 172.16.118.92:53] client/internal/dns/upstream.go:91: got an error while connecting to upstream
2024-02-10T19:18:23-03:00 WARN [error: read udp 192.168.3.132:42452->172.16.118.92:53: i/o timeout, upstream: 172.16.118.92:53] client/internal/dns/upstream.go:91: got an error while connecting to upstream
2024-02-10T19:18:23-03:00 ERRO client/internal/dns/upstream.go:123: all queries to the upstream nameservers failed with timeout
2024-02-10T19:18:23-03:00 WARN [error: read udp 192.168.3.132:52209->172.16.118.92:53: i/o timeout, upstream: 172.16.118.92:53] client/internal/dns/upstream.go:91: got an error while connecting to upstream
2024-02-10T19:18:23-03:00 ERRO client/internal/dns/upstream.go:123: all queries to the upstream nameservers failed with timeout
2024-02-10T19:18:23-03:00 WARN [error: read udp 192.168.3.132:60433->172.16.118.92:53: i/o timeout, upstream: 172.16.118.92:53] client/internal/dns/upstream.go:91: got an error while connecting to upstream
2024-02-10T19:18:23-03:00 ERRO client/internal/dns/upstream.go:123: all queries to the upstream nameservers failed with timeout
2024-02-10T19:18:23-03:00 WARN [error: read udp 192.168.3.132:60661->172.16.118.92:53: i/o timeout, upstream: 172.16.118.92:53] client/internal/dns/upstream.go:91: got an error while connecting to upstream
2024-02-10T19:18:23-03:00 ERRO client/internal/dns/upstream.go:123: all queries to the upstream nameservers failed with timeout
2024-02-10T19:18:23-03:00 WARN [error: read udp 192.168.3.132:36855->172.16.118.92:53: i/o timeout, upstream: 172.16.118.92:53] client/internal/dns/upstream.go:91: got an error while connecting to upstream
2024-02-10T19:18:23-03:00 ERRO client/internal/dns/upstream.go:123: all queries to the upstream nameservers failed with timeout
2024-02-10T19:18:23-03:00 WARN [error: read udp 192.168.3.132:47221->172.16.118.92:53: i/o timeout, upstream: 172.16.118.92:53] client/internal/dns/upstream.go:91: got an error while connecting to upstream
2024-02-10T19:18:23-03:00 ERRO client/internal/dns/upstream.go:123: all queries to the upstream nameservers failed with timeout
2024-02-10T19:18:23-03:00 ERRO client/internal/dns/upstream.go:123: all queries to the upstream nameservers failed with timeout
2024-02-10T19:18:23-03:00 WARN [error: read udp 192.168.3.132:53991->172.16.118.92:53: i/o timeout, upstream: 172.16.118.92:53] client/internal/dns/upstream.go:91: got an error while connecting to upstream
2024-02-10T19:18:23-03:00 ERRO client/internal/dns/upstream.go:123: all queries to the upstream nameservers failed with timeout
2024-02-10T19:18:26-03:00 WARN [error: read udp 100.89.144.116:40182->172.16.118.92:53: i/o timeout, upstream: 172.16.118.92:53] client/internal/dns/upstream.go:91: got an error while connecting to upstream
2024-02-10T19:18:26-03:00 ERRO client/internal/dns/upstream.go:123: all queries to the upstream nameservers failed with timeout
2024-02-10T19:18:26-03:00 WARN [error: read udp 100.89.144.116:48873->172.16.118.92:53: i/o timeout, upstream: 172.16.118.92:53] client/internal/dns/upstream.go:91: got an error while connecting to upstream
2024-02-10T19:18:26-03:00 ERRO client/internal/dns/upstream.go:123: all queries to the upstream nameservers failed with timeout
2024-02-10T19:18:26-03:00 WARN [error: read udp 100.89.144.116:49989->172.16.118.92:53: i/o timeout, upstream: 172.16.118.92:53] client/internal/dns/upstream.go:91: got an error while connecting to upstream
2024-02-10T19:18:26-03:00 WARN [error: read udp 100.89.144.116:48949->172.16.118.92:53: i/o timeout, upstream: 172.16.118.92:53] client/internal/dns/upstream.go:91: got an error while connecting to upstream
2024-02-10T19:18:26-03:00 ERRO client/internal/dns/upstream.go:123: all queries to the upstream nameservers failed with timeout
2024-02-10T19:18:26-03:00 ERRO client/internal/dns/upstream.go:123: all queries to the upstream nameservers failed with timeout
2024-02-10T19:18:26-03:00 WARN [upstream: 172.16.118.92:53, error: read udp 100.89.144.116:49487->172.16.118.92:53: i/o timeout] client/internal/dns/upstream.go:91: got an error while connecting to upstream
2024-02-10T19:18:26-03:00 ERRO client/internal/dns/upstream.go:123: all queries to the upstream nameservers failed with timeout
2024-02-10T19:18:26-03:00 WARN [upstream: 172.16.118.92:53, error: read udp 100.89.144.116:33265->172.16.118.92:53: i/o timeout] client/internal/dns/upstream.go:91: got an error while connecting to upstream
2024-02-10T19:18:26-03:00 ERRO client/internal/dns/upstream.go:123: all queries to the upstream nameservers failed with timeout
2024-02-10T19:18:26-03:00 WARN [upstream: 172.16.118.92:53, error: read udp 100.89.144.116:58250->172.16.118.92:53: i/o timeout] client/internal/dns/upstream.go:91: got an error while connecting to upstream
2024-02-10T19:18:26-03:00 ERRO client/internal/dns/upstream.go:123: all queries to the upstream nameservers failed with timeout
2024-02-10T19:18:26-03:00 WARN [error: read udp 100.89.144.116:48172->172.16.118.92:53: i/o timeout, upstream: 172.16.118.92:53] client/internal/dns/upstream.go:91: got an error while connecting to upstream
2024-02-10T19:18:26-03:00 ERRO client/internal/dns/upstream.go:123: all queries to the upstream nameservers failed with timeout
2024-02-10T19:18:27-03:00 INFO client/internal/dns/upstream.go:227: upstreams [172.16.118.92:53] are responsive again. Adding them back to system
2024-02-10T19:18:27-03:00 DEBG [nameservers: [{172.16.118.92 udp 53}]] client/internal/dns/server.go:505: reactivate temporary disabled nameserver group
2024-02-10T19:18:27-03:00 INFO client/internal/dns/network_manager_linux.go:140: configured 100.89.144.116:53 as main DNS forwarder for this peer
2024-02-10T19:18:27-03:00 INFO client/internal/dns/network_manager_linux.go:160: adding 1 search domains and 0 match domains. Search list: [netbird.HOST.] , Match list: []
2024-02-10T19:18:31-03:00 WARN [error: read udp 192.168.3.132:56029->172.16.118.92:53: i/o timeout, upstream: 172.16.118.92:53] client/internal/dns/upstream.go:91: got an error while connecting to upstream
2024-02-10T19:18:31-03:00 ERRO client/internal/dns/upstream.go:123: all queries to the upstream nameservers failed with timeout
2024-02-10T19:18:31-03:00 WARN [upstream: 172.16.118.92:53, error: read udp 192.168.3.132:44808->172.16.118.92:53: i/o timeout] client/internal/dns/upstream.go:91: got an error while connecting to upstream
2024-02-10T19:18:31-03:00 ERRO client/internal/dns/upstream.go:123: all queries to the upstream nameservers failed with timeout
2024-02-10T19:18:31-03:00 WARN [error: read udp 192.168.3.132:40947->172.16.118.92:53: i/o timeout, upstream: 172.16.118.92:53] client/internal/dns/upstream.go:91: got an error while connecting to upstream
2024-02-10T19:18:31-03:00 ERRO client/internal/dns/upstream.go:123: all queries to the upstream nameservers failed with timeout
2024-02-10T19:18:31-03:00 WARN [error: read udp 192.168.3.132:57774->172.16.118.92:53: i/o timeout, upstream: 172.16.118.92:53] client/internal/dns/upstream.go:91: got an error while connecting to upstream
2024-02-10T19:18:31-03:00 ERRO client/internal/dns/upstream.go:123: all queries to the upstream nameservers failed with timeout
2024-02-10T19:18:31-03:00 WARN [error: read udp 192.168.3.132:37716->172.16.118.92:53: i/o timeout, upstream: 172.16.118.92:53] client/internal/dns/upstream.go:91: got an error while connecting to upstream
2024-02-10T19:18:31-03:00 ERRO client/internal/dns/upstream.go:123: all queries to the upstream nameservers failed with timeout
2024-02-10T19:18:31-03:00 WARN client/internal/dns/upstream.go:251: upstream resolving is Disabled for 30s
2024-02-10T19:18:31-03:00 INFO [nameservers: [{172.16.118.92 udp 53}]] client/internal/dns/server.go:470: temporary deactivate nameservers group due timeout
2024-02-10T19:18:31-03:00 WARN [error: read udp 192.168.3.132:35823->172.16.118.92:53: i/o timeout, upstream: 172.16.118.92:53] client/internal/dns/upstream.go:91: got an error while connecting to upstream
2024-02-10T19:18:31-03:00 ERRO client/internal/dns/upstream.go:123: all queries to the upstream nameservers failed with timeout
2024-02-10T19:18:31-03:00 WARN [upstream: 172.16.118.92:53, error: read udp 192.168.3.132:51648->172.16.118.92:53: i/o timeout] client/internal/dns/upstream.go:91: got an error while connecting to upstream
2024-02-10T19:18:31-03:00 ERRO client/internal/dns/upstream.go:123: all queries to the upstream nameservers failed with timeout
2024-02-10T19:18:31-03:00 WARN [error: read udp 192.168.3.132:47458->172.16.118.92:53: i/o timeout, upstream: 172.16.118.92:53] client/internal/dns/upstream.go:91: got an error while connecting to upstream
2024-02-10T19:18:31-03:00 ERRO client/internal/dns/upstream.go:123: all queries to the upstream nameservers failed with timeout
2024-02-10T19:18:31-03:00 INFO client/internal/dns/network_manager_linux.go:160: adding 1 search domains and 0 match domains. Search list: [netbird.HOST.] , Match list: []
jumski commented 4 months ago

I think i suffer from the same bug. I believe there is some kind of problem with managing /etc/resolv.conf when other process is also managing it (NetworkManager in my case).

This also manifests in the log output pasted by @eriverosr - in the first, correct one, you can grep for resolve.conf but the second log does not have entries about watching/managing this file.

The bug is extremely annoying :-(

jumski commented 4 months ago

This problem resolved itself on my end. I'm not sure if this was caused by recent update or my doings, but this is roughly what i did (maybe will be useful for someone):