search

netbirdio / netbird

Connect your devices into a secure WireGuard®-based overlay network with SSO, MFA and granular access controls.
https://netbird.io
BSD 3-Clause "New" or "Revised" License
11.07k stars 509 forks source link

macOS client does not add routes properly #1900

Open damlys opened 6 months ago

damlys commented 6 months ago

Describe the problem

It seems that macOS client does not add routes properly. Some of the traffic is not routed by NetBird peers.

To Reproduce

(IP addresses were modified for this example)

Steps to reproduce the behavior:

  1. Add new docker peer
  2. Add following routes 34.116.146.33/32, 34.116.162.131/32, 34.118.106.37/32, 34.118.116.195/32 (they are internet adesses) and set routing peer to use previous added one

  3. Observe /var/log/netbird/client.log file

      --- CONNECT

  2024-04-29T15:08:52+02:00 WARN client/internal/routemanager/client.go:153: the network 34.118.106.37/32 has not been assigned a routing peer as no peers from the list [KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=] are currently connected
  2024-04-29T15:08:52+02:00 INFO client/internal/dns/host_darwin.go:157: added 1 search domains to the state. Domain list: netbird.selfhosted
  2024-04-29T15:08:52+02:00 INFO client/internal/acl/manager.go:52: ACL rules processed in: 108µs, total rules count: 2
  2024-04-29T15:09:08+02:00 ERRO util/net/dialer_generic.go:64: Failed to call dialer hooks: failed to resolve address :51820: lookup : no such host
  2024-04-29T15:09:08+02:00 ERRO client/internal/peer/conn.go:444: Before add peer hook failed: adding route reference: failed to add route for prefix 127.0.0.1/32: route not allowed
  2024-04-29T15:09:08+02:00 INFO client/internal/peer/conn.go:388: connected to peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=, endpoint address: 127.0.0.1:53675
  2024-04-29T15:09:08+02:00 INFO client/internal/routemanager/client.go:158: new chosen route is cokalj9mjugs73f8kfbg with peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4= with score 0.964561 for network 34.118.116.195/32
  2024-04-29T15:09:08+02:00 INFO client/internal/routemanager/client.go:158: new chosen route is cojqhj1mjugs73f8kf7g with peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4= with score 0.964561 for network 34.116.146.33/32
  2024-04-29T15:09:08+02:00 INFO client/internal/routemanager/client.go:158: new chosen route is cojqhopmjugs73f8kf80 with peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4= with score 0.964561 for network 34.118.106.37/32
  2024-04-29T15:09:08+02:00 INFO client/internal/routemanager/client.go:158: new chosen route is cokal7pmjugs73f8kfb0 with peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4= with score 0.964561 for network 34.116.162.131/32

  --- DISCONNECT

  2024-04-29T15:16:45+02:00 ERRO client/internal/routemanager/client.go:318: Couldn't remove route from peer and system for network 34.116.146.33/32: remove route: get peer state: peer not found
  2024-04-29T15:16:45+02:00 ERRO client/internal/routemanager/client.go:318: Couldn't remove route from peer and system for network 34.116.162.131/32: remove route: get peer state: peer not found
  2024-04-29T15:16:45+02:00 ERRO client/internal/routemanager/client.go:318: Couldn't remove route from peer and system for network 34.118.116.195/32: remove route: get peer state: peer not found
  2024-04-29T15:16:45+02:00 ERRO client/internal/routemanager/client.go:318: Couldn't remove route from peer and system for network 34.118.106.37/32: remove route: get peer state: peer not found
  2024-04-29T15:16:46+02:00 INFO client/internal/dns/host_darwin.go:120: removing search domains from system
  2024-04-29T15:16:46+02:00 INFO client/internal/routemanager/manager.go:112: Routing cleanup complete
  2024-04-29T15:16:46+02:00 INFO client/internal/engine.go:224: stopped Netbird Engine
  2024-04-29T15:16:46+02:00 INFO client/internal/connect.go:261: stopped NetBird client

  --- CONNECT

  2024-04-29T15:17:12+02:00 INFO client/internal/connect.go:104: starting NetBird client version 0.27.3 on darwin/arm64
  2024-04-29T15:17:14+02:00 INFO client/internal/routemanager/manager.go:88: Routing setup complete
  2024-04-29T15:17:14+02:00 INFO signal/client/grpc.go:158: connected to the Signal Service stream
  2024-04-29T15:17:14+02:00 INFO client/internal/connect.go:247: Netbird engine started, my IP is: 100.64.240.231/16
  2024-04-29T15:17:14+02:00 INFO management/client/grpc.go:147: connected to the Management Service stream
  2024-04-29T15:17:14+02:00 WARN client/internal/engine.go:507: running SSH server is not permitted
  2024-04-29T15:17:14+02:00 WARN client/internal/routemanager/client.go:153: the network 34.118.106.37/32 has not been assigned a routing peer as no peers from the list [KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=] are currently connected
  2024-04-29T15:17:14+02:00 WARN client/internal/routemanager/client.go:153: the network 34.118.116.195/32 has not been assigned a routing peer as no peers from the list [KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=] are currently connected
  2024-04-29T15:17:14+02:00 WARN client/internal/routemanager/client.go:153: the network 34.116.162.131/32 has not been assigned a routing peer as no peers from the list [KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=] are currently connected
  2024-04-29T15:17:14+02:00 WARN client/internal/routemanager/client.go:153: the network 34.116.146.33/32 has not been assigned a routing peer as no peers from the list [KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=] are currently connected
  2024-04-29T15:17:14+02:00 INFO client/internal/dns/host_darwin.go:91: removing match domains from the system
  2024-04-29T15:17:14+02:00 INFO client/internal/dns/host_darwin.go:157: added 1 search domains to the state. Domain list: netbird.selfhosted
  2024-04-29T15:17:14+02:00 INFO client/internal/acl/manager.go:52: ACL rules processed in: 73.041µs, total rules count: 2
  2024-04-29T15:17:17+02:00 ERRO util/net/dialer_generic.go:64: Failed to call dialer hooks: failed to resolve address :51820: lookup : no such host
  2024-04-29T15:17:17+02:00 ERRO client/internal/peer/conn.go:444: Before add peer hook failed: adding route reference: failed to add route for prefix 127.0.0.1/32: route not allowed
  2024-04-29T15:17:17+02:00 INFO client/internal/peer/conn.go:388: connected to peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=, endpoint address: 127.0.0.1:60859
  2024-04-29T15:17:17+02:00 INFO client/internal/routemanager/client.go:158: new chosen route is cokal7pmjugs73f8kfb0 with peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4= with score 0.983135 for network 34.116.162.131/32
  2024-04-29T15:17:17+02:00 INFO client/internal/routemanager/client.go:158: new chosen route is cojqhopmjugs73f8kf80 with peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4= with score 0.983135 for network 34.118.106.37/32
  2024-04-29T15:17:17+02:00 INFO client/internal/routemanager/client.go:158: new chosen route is cokalj9mjugs73f8kfbg with peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4= with score 0.983135 for network 34.118.116.195/32
  2024-04-29T15:17:17+02:00 INFO client/internal/routemanager/client.go:158: new chosen route is cojqhj1mjugs73f8kf7g with peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4= with score 0.983135 for network 34.116.146.33/32

  --- DISCONNECT

  2024-04-29T15:17:45+02:00 ERRO client/internal/routemanager/client.go:318: Couldn't remove route from peer and system for network 34.116.146.33/32: remove route: get peer state: peer not found
  2024-04-29T15:17:45+02:00 ERRO client/internal/routemanager/client.go:318: Couldn't remove route from peer and system for network 34.118.116.195/32: remove route: get peer state: peer not found
  2024-04-29T15:17:45+02:00 ERRO client/internal/routemanager/client.go:318: Couldn't remove route from peer and system for network 34.116.162.131/32: remove route: get peer state: peer not found
  2024-04-29T15:17:45+02:00 ERRO client/internal/routemanager/client.go:318: Couldn't remove route from peer and system for network 34.118.106.37/32: remove route: get peer state: peer not found
  2024-04-29T15:17:46+02:00 INFO client/internal/dns/host_darwin.go:120: removing search domains from system
  2024-04-29T15:17:46+02:00 INFO client/internal/routemanager/manager.go:112: Routing cleanup complete
  2024-04-29T15:17:46+02:00 INFO client/internal/engine.go:224: stopped Netbird Engine
  2024-04-29T15:17:46+02:00 INFO client/internal/connect.go:261: stopped NetBird client

  --- CONNECT

  2024-04-29T15:18:11+02:00 INFO client/internal/connect.go:104: starting NetBird client version 0.27.3 on darwin/arm64
  2024-04-29T15:18:11+02:00 INFO client/internal/routemanager/manager.go:88: Routing setup complete
  2024-04-29T15:18:12+02:00 INFO signal/client/grpc.go:158: connected to the Signal Service stream
  2024-04-29T15:18:12+02:00 INFO client/internal/connect.go:247: Netbird engine started, my IP is: 100.64.240.231/16
  2024-04-29T15:18:12+02:00 INFO management/client/grpc.go:147: connected to the Management Service stream
  2024-04-29T15:18:12+02:00 WARN client/internal/engine.go:507: running SSH server is not permitted
  2024-04-29T15:18:12+02:00 WARN client/internal/routemanager/client.go:153: the network 34.116.162.131/32 has not been assigned a routing peer as no peers from the list [KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=] are currently connected
  2024-04-29T15:18:12+02:00 WARN client/internal/routemanager/client.go:153: the network 34.116.146.33/32 has not been assigned a routing peer as no peers from the list [KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=] are currently connected
  2024-04-29T15:18:12+02:00 WARN client/internal/routemanager/client.go:153: the network 34.118.116.195/32 has not been assigned a routing peer as no peers from the list [KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=] are currently connected
  2024-04-29T15:18:12+02:00 WARN client/internal/routemanager/client.go:153: the network 34.118.106.37/32 has not been assigned a routing peer as no peers from the list [KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=] are currently connected
  2024-04-29T15:18:12+02:00 INFO client/internal/dns/host_darwin.go:91: removing match domains from the system
  2024-04-29T15:18:12+02:00 INFO client/internal/dns/host_darwin.go:157: added 1 search domains to the state. Domain list: netbird.selfhosted
  2024-04-29T15:18:12+02:00 INFO client/internal/acl/manager.go:52: ACL rules processed in: 65µs, total rules count: 2
  2024-04-29T15:18:14+02:00 ERRO util/net/dialer_generic.go:64: Failed to call dialer hooks: failed to resolve address :51820: lookup : no such host
  2024-04-29T15:18:14+02:00 ERRO client/internal/peer/conn.go:444: Before add peer hook failed: adding route reference: failed to add route for prefix 127.0.0.1/32: route not allowed
  2024-04-29T15:18:14+02:00 INFO client/internal/peer/conn.go:388: connected to peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=, endpoint address: 127.0.0.1:56858
  2024-04-29T15:18:14+02:00 INFO client/internal/routemanager/client.go:158: new chosen route is cokal7pmjugs73f8kfb0 with peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4= with score 0.986417 for network 34.116.162.131/32
  2024-04-29T15:18:14+02:00 INFO client/internal/routemanager/client.go:158: new chosen route is cokalj9mjugs73f8kfbg with peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4= with score 0.986417 for network 34.118.116.195/32
  2024-04-29T15:18:14+02:00 INFO client/internal/routemanager/client.go:158: new chosen route is cojqhopmjugs73f8kf80 with peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4= with score 0.986417 for network 34.118.106.37/32
  2024-04-29T15:18:14+02:00 INFO client/internal/routemanager/client.go:158: new chosen route is cojqhj1mjugs73f8kf7g with peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4= with score 0.986417 for network 34.116.146.33/32
  2024-04-29T15:18:14+02:00 WARN client/internal/routemanager/systemops.go:259: Skipping adding a new route for network 34.118.106.37/32 because it already exists
  4. See route does not exist 
    $ netstat -rn | grep '34.'
34.116.146.33      utun100            UHS               utun100
34.116.162.131     utun100            UHS               utun100
34.118.77.27       192.168.33.1       UGHS                  en0
34.118.116.195     utun100            UHS               utun100

Expected behavior

All routes exists on my macOS machine.

Are you using NetBird Cloud?

self-host

NetBird version

$ netbird version
0.27.3

NetBird status -d output:

(host was modified for this example)

$ netbird status --detail
Peers detail:
 vm-iceo-netbird-1-europe-central2-a-main.netbird.selfhosted:
  NetBird IP: 100.64.192.223
  Public key: KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=
  Status: Connected
  -- detail --
  Connection type: Relayed
  Direct: false
  ICE candidate (Local/Remote): relay/host
  ICE candidate endpoints (Local/Remote): 34.118.77.27:56882/172.17.0.2:56882
  Last connection update: 2024-04-29 15:18:14
  Last WireGuard handshake: 2024-04-29 15:21:06
  Transfer status (received/sent) 2.0 KiB/780 B
  Quantum resistance: false
  Routes: 34.116.146.33/32, 34.116.162.131/32, 34.118.106.37/32, 34.118.116.195/32
  Latency: 16.180875ms

Daemon version: 0.27.3
CLI version: 0.27.3
Management: Connected to https://netbird.example.com:33073
Signal: Connected to http://netbird.example.com:10000
Relays:
  [stun:netbird.example.com:3478] is Available
  [turn:netbird.example.com:3478?transport=udp] is Available
Nameservers:
FQDN: damians-macbook-pro.netbird.selfhosted
NetBird IP: 100.64.240.231/16
Interface type: Userspace
Quantum resistance: false
Routes: -
Peers count: 1/1 Connected
lixmal commented 6 months ago

@damlys Is the issue reproducible for you? I cannot reproduce it myself

Could you provide more of the logs on debug level, please?

https://docs.netbird.io/how-to/troubleshooting-client#enabling-debug-logs-on-agent or netbird debug for 1m -A on v0.27.4

velia-fehrenthal commented 6 months ago

I think I just experienced the same issue. turns out netstat -rna did show a cloned route which prevented netbird from adding or deleting its intended route. adding the route manually did succeed and traffic was flowing as expected

lixmal commented 6 months ago

@damlys if you still have that issue, can you see if you can find the route in netstat -nra?

ergleb78 commented 6 months ago

@lixmal I'm facing similar issue on MacOS 14.4.1 (23E224) Steps to reproduce:

  1. Install netmaker on MacOS.
  2. Add routes to external node
  3. Everything is working fine, required routes are in the routing table.
  4. Disconnect netmaker
  5. Connect netmaker (all peers discovered properly)
  6. Try running netstat -nr => no route in the table
damlys commented 6 months ago

@velia-fehrenthal indeed, it seems that routes are within netstat -nra. There are some logs

--- ROUTES

$ netstat -rn | grep '34.'

$ netstat -nra | grep '34.'
34.107.204.85      192.168.33.1       UGHWIig               en0
34.116.146.33      192.168.33.1       UGHWIig               en0
34.116.162.131     192.168.33.1       UGHWIig               en0

--- CONNECT

2024-05-08T09:28:42+02:00 WARN client/server/server.go:256: failed login: rpc error: code = InvalidArgument desc = invalid setup-key or no sso information provided, err: invalid UUID length: 0
2024-05-08T09:28:47+02:00 INFO client/internal/login.go:130: peer has been successfully registered on Management Service
2024-05-08T09:28:47+02:00 INFO client/internal/connect.go:104: starting NetBird client version 0.27.3 on darwin/arm64
2024-05-08T09:28:48+02:00 INFO client/internal/routemanager/manager.go:88: Routing setup complete
2024-05-08T09:28:48+02:00 INFO client/internal/connect.go:247: Netbird engine started, my IP is: 100.64.240.231/16
2024-05-08T09:28:48+02:00 INFO signal/client/grpc.go:158: connected to the Signal Service stream
2024-05-08T09:28:48+02:00 INFO management/client/grpc.go:147: connected to the Management Service stream
2024-05-08T09:28:48+02:00 WARN client/internal/engine.go:507: running SSH server is not permitted
2024-05-08T09:28:48+02:00 WARN client/internal/routemanager/client.go:153: the network 34.118.116.195/32 has not been assigned a routing peer as no peers from the list [KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=] are currently connected
2024-05-08T09:28:48+02:00 WARN client/internal/routemanager/client.go:153: the network 34.116.162.131/32 has not been assigned a routing peer as no peers from the list [KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=] are currently connected
2024-05-08T09:28:48+02:00 WARN client/internal/routemanager/client.go:153: the network 34.117.118.44/32 has not been assigned a routing peer as no peers from the list [KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=] are currently connected
2024-05-08T09:28:48+02:00 WARN client/internal/routemanager/client.go:153: the network 34.118.106.37/32 has not been assigned a routing peer as no peers from the list [KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=] are currently connected
2024-05-08T09:28:48+02:00 WARN client/internal/routemanager/client.go:153: the network 34.116.146.33/32 has not been assigned a routing peer as no peers from the list [KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=] are currently connected
2024-05-08T09:28:48+02:00 INFO client/internal/dns/host_darwin.go:91: removing match domains from the system
2024-05-08T09:28:48+02:00 INFO client/internal/dns/host_darwin.go:157: added 1 search domains to the state. Domain list: netbird.selfhosted
2024-05-08T09:28:48+02:00 INFO client/internal/acl/manager.go:52: ACL rules processed in: 310.792µs, total rules count: 2
2024-05-08T09:28:51+02:00 ERRO util/net/dialer_generic.go:64: Failed to call dialer hooks: failed to resolve address :51820: lookup : no such host
2024-05-08T09:28:51+02:00 ERRO client/internal/peer/conn.go:444: Before add peer hook failed: adding route reference: failed to add route for prefix 127.0.0.1/32: route not allowed
2024-05-08T09:28:51+02:00 INFO client/internal/peer/conn.go:388: connected to peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=, endpoint address: 127.0.0.1:52034
2024-05-08T09:28:51+02:00 INFO client/internal/routemanager/client.go:158: new chosen route is cokalj9mjugs73f8kfbg with peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4= with score 0.980318 for network 34.118.116.195/32
2024-05-08T09:28:51+02:00 INFO client/internal/routemanager/client.go:158: new chosen route is cojqhopmjugs73f8kf80 with peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4= with score 0.980318 for network 34.118.106.37/32
2024-05-08T09:28:51+02:00 INFO client/internal/routemanager/client.go:158: new chosen route is cojqhj1mjugs73f8kf7g with peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4= with score 0.980318 for network 34.116.146.33/32
2024-05-08T09:28:51+02:00 WARN client/internal/routemanager/systemops.go:259: Skipping adding a new route for network 34.118.106.37/32 because it already exists
2024-05-08T09:28:51+02:00 INFO client/internal/routemanager/client.go:158: new chosen route is cokal7pmjugs73f8kfb0 with peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4= with score 0.980318 for network 34.116.162.131/32
2024-05-08T09:28:51+02:00 WARN client/internal/routemanager/systemops.go:259: Skipping adding a new route for network 34.116.146.33/32 because it already exists
2024-05-08T09:28:51+02:00 INFO client/internal/routemanager/client.go:158: new chosen route is coof3gpmjugs73f8kgig with peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4= with score 99980.980318 for network 34.117.118.44/32
2024-05-08T09:28:51+02:00 WARN client/internal/routemanager/systemops.go:259: Skipping adding a new route for network 34.116.162.131/32 because it already exists

--- ROUTES

$ netstat -rn | grep '34.'
34.117.118.44      utun100            UHS               utun100
34.118.77.27       192.168.33.1       UGHS                  en0
34.118.116.195     utun100            UHS               utun100

$ netstat -nra | grep '34.'
34.116.146.33      192.168.33.1       UGHWIig               en0
34.116.162.131     192.168.33.1       UGHWIig               en0
34.117.118.44      utun100            UHS               utun100
34.118.77.27       192.168.33.1       UGHS                  en0
34.118.77.27       192.168.33.1       UGHWIig               en0
34.118.116.195     utun100            UHS               utun100

--- DISCONNECT

2024-05-08T09:31:38+02:00 ERRO client/internal/routemanager/client.go:318: Couldn't remove route from peer and system for network 34.117.118.44/32: remove route: get peer state: peer not found
2024-05-08T09:31:38+02:00 ERRO client/internal/routemanager/client.go:318: Couldn't remove route from peer and system for network 34.116.146.33/32: remove route: get peer state: peer not found
2024-05-08T09:31:38+02:00 ERRO client/internal/routemanager/client.go:318: Couldn't remove route from peer and system for network 34.116.162.131/32: remove route: get peer state: peer not found
2024-05-08T09:31:38+02:00 ERRO client/internal/routemanager/client.go:318: Couldn't remove route from peer and system for network 34.118.106.37/32: remove route: get peer state: peer not found
2024-05-08T09:31:38+02:00 ERRO client/internal/routemanager/client.go:318: Couldn't remove route from peer and system for network 34.118.116.195/32: remove route: get peer state: peer not found
2024-05-08T09:31:38+02:00 INFO client/internal/dns/host_darwin.go:120: removing search domains from system
2024-05-08T09:31:38+02:00 INFO client/internal/routemanager/manager.go:112: Routing cleanup complete
2024-05-08T09:31:38+02:00 INFO client/internal/engine.go:224: stopped Netbird Engine
2024-05-08T09:31:38+02:00 INFO client/internal/connect.go:261: stopped NetBird client

--- ROUTES

$ netstat -rn | grep '34.'

$ netstat -nra | grep '34.'
34.116.146.33      192.168.33.1       UGHWIig               en0
34.116.162.131     192.168.33.1       UGHWIig               en0
34.118.77.27       192.168.33.1       UGHWIig               en0

--- CONNECT

2024-05-08T09:33:30+02:00 INFO client/internal/connect.go:104: starting NetBird client version 0.27.3 on darwin/arm64
2024-05-08T09:33:31+02:00 INFO client/internal/routemanager/manager.go:88: Routing setup complete
2024-05-08T09:33:31+02:00 INFO signal/client/grpc.go:158: connected to the Signal Service stream
2024-05-08T09:33:31+02:00 INFO client/internal/connect.go:247: Netbird engine started, my IP is: 100.64.240.231/16
2024-05-08T09:33:31+02:00 INFO management/client/grpc.go:147: connected to the Management Service stream
2024-05-08T09:33:31+02:00 WARN client/internal/engine.go:507: running SSH server is not permitted
2024-05-08T09:33:31+02:00 WARN client/internal/routemanager/client.go:153: the network 34.118.106.37/32 has not been assigned a routing peer as no peers from the list [KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=] are currently connected
2024-05-08T09:33:31+02:00 WARN client/internal/routemanager/client.go:153: the network 34.118.116.195/32 has not been assigned a routing peer as no peers from the list [KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=] are currently connected
2024-05-08T09:33:31+02:00 WARN client/internal/routemanager/client.go:153: the network 34.117.118.44/32 has not been assigned a routing peer as no peers from the list [KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=] are currently connected
2024-05-08T09:33:31+02:00 WARN client/internal/routemanager/client.go:153: the network 34.116.146.33/32 has not been assigned a routing peer as no peers from the list [KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=] are currently connected
2024-05-08T09:33:31+02:00 INFO client/internal/dns/host_darwin.go:91: removing match domains from the system
2024-05-08T09:33:31+02:00 WARN client/internal/routemanager/client.go:153: the network 34.116.162.131/32 has not been assigned a routing peer as no peers from the list [KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=] are currently connected
2024-05-08T09:33:31+02:00 INFO client/internal/dns/host_darwin.go:157: added 1 search domains to the state. Domain list: netbird.selfhosted
2024-05-08T09:33:31+02:00 INFO client/internal/acl/manager.go:52: ACL rules processed in: 153.666µs, total rules count: 2
2024-05-08T09:33:34+02:00 ERRO util/net/dialer_generic.go:64: Failed to call dialer hooks: failed to resolve address :51820: lookup : no such host
2024-05-08T09:33:34+02:00 ERRO client/internal/peer/conn.go:444: Before add peer hook failed: adding route reference: failed to add route for prefix 127.0.0.1/32: route not allowed
2024-05-08T09:33:34+02:00 INFO client/internal/peer/conn.go:388: connected to peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=, endpoint address: 127.0.0.1:60227
2024-05-08T09:33:34+02:00 INFO client/internal/routemanager/client.go:158: new chosen route is cokalj9mjugs73f8kfbg with peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4= with score 0.979084 for network 34.118.116.195/32
2024-05-08T09:33:34+02:00 INFO client/internal/routemanager/client.go:158: new chosen route is cojqhj1mjugs73f8kf7g with peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4= with score 0.979084 for network 34.116.146.33/32
2024-05-08T09:33:34+02:00 WARN client/internal/routemanager/systemops.go:259: Skipping adding a new route for network 34.116.146.33/32 because it already exists
2024-05-08T09:33:34+02:00 INFO client/internal/routemanager/client.go:158: new chosen route is cojqhopmjugs73f8kf80 with peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4= with score 0.979084 for network 34.118.106.37/32
2024-05-08T09:33:34+02:00 INFO client/internal/routemanager/client.go:158: new chosen route is coof3gpmjugs73f8kgig with peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4= with score 99980.979084 for network 34.117.118.44/32
2024-05-08T09:33:34+02:00 INFO client/internal/routemanager/client.go:158: new chosen route is cokal7pmjugs73f8kfb0 with peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4= with score 0.979084 for network 34.116.162.131/32
2024-05-08T09:33:34+02:00 WARN client/internal/routemanager/systemops.go:259: Skipping adding a new route for network 34.116.162.131/32 because it already exists

--- ROUTES

$ netstat -rn | grep '34.'
34.117.118.44      utun100            UHS               utun100
34.118.77.27       192.168.33.1       UGHS                  en0
34.118.106.37      utun100            UHS               utun100
34.118.116.195     utun100            UHS               utun100

$ netstat -nra | grep '34.'
34.116.146.33      192.168.33.1       UGHWIig               en0
34.116.162.131     192.168.33.1       UGHWIig               en0
34.117.118.44      utun100            UHS               utun100
34.118.77.27       192.168.33.1       UGHS                  en0
34.118.77.27       192.168.33.1       UGHWIig               en0
34.118.106.37      utun100            UHS               utun100
34.118.116.195     utun100            UHS               utun100

@lixmal and the same with debug level 👀 

--- ROUTES

$ netstat -rn | grep '34\.'

$ netstat -nra | grep '34\.'
34.116.146.33      192.168.33.1       UGHWIig               en0
34.116.162.131     192.168.33.1       UGHWIig               en0
34.118.77.27       192.168.33.1       UGHWIig               en0

--- CONNECT

2024-05-08T09:59:01+02:00 DEBG client/internal/login.go:93: connecting to the Management service https://netbird.iceolab.com:33073
2024-05-08T09:59:01+02:00 DEBG client/internal/login.go:63: connected to the Management service https://netbird.iceolab.com:33073
2024-05-08T09:59:03+02:00 INFO client/internal/connect.go:104: starting NetBird client version 0.27.3 on darwin/arm64
2024-05-08T09:59:03+02:00 DEBG client/internal/connect.go:165: connecting to the Management service netbird.iceolab.com:33073
2024-05-08T09:59:03+02:00 DEBG client/internal/connect.go:173: connected to the Management service netbird.iceolab.com:33073
2024-05-08T09:59:04+02:00 DEBG signal/client/grpc.go:92: connected to Signal Service: netbird.iceolab.com:10000
2024-05-08T09:59:04+02:00 DEBG client/internal/routemanager/systemops.go:87: Route for 0.0.0.0: interface &{24 1500 bridge100 b2:de:28:e8:25:64 up|broadcast|multicast|running}, nexthop <nil>, preferred source 192.168.64.1
2024-05-08T09:59:04+02:00 DEBG client/internal/routemanager/systemops.go:92: No next hop found for ip 0.0.0.0, using preferred source 192.168.64.1
2024-05-08T09:59:04+02:00 DEBG client/internal/routemanager/systemops.go:87: Route for ::: interface &{28 1380 utun8  up|pointtopoint|multicast|running}, nexthop fe80::, preferred source fe80::5411:46d9:db4e:a562
2024-05-08T09:59:04+02:00 DEBG client/internal/routemanager/routemanager.go:51: Increasing route ref count 0 for prefix 34.118.77.27/32
2024-05-08T09:59:04+02:00 DEBG client/internal/routemanager/routemanager.go:55: Adding route for prefix 34.118.77.27/32
2024-05-08T09:59:04+02:00 DEBG client/internal/routemanager/systemops.go:87: Route for 34.118.77.27: interface &{15 1500 en0 b0:de:28:8e:3f:d6 up|broadcast|multicast|running}, nexthop 192.168.33.1, preferred source 192.168.33.3
2024-05-08T09:59:04+02:00 DEBG client/internal/routemanager/systemops.go:180: Found next hop 192.168.33.1 for prefix 34.118.77.27/32 with interface &{15 1500 en0 b0:de:28:8e:3f:d6 up|broadcast|multicast|running}
2024-05-08T09:59:04+02:00 DEBG client/internal/routemanager/systemops.go:201: Adding a new route for prefix 34.118.77.27/32 with next hop 192.168.33.1
2024-05-08T09:59:04+02:00 DEBG client/internal/routemanager/routemanager.go:51: Increasing route ref count 0 for prefix 2600:1900:4140:4163:0:a::/128
2024-05-08T09:59:04+02:00 DEBG client/internal/routemanager/routemanager.go:55: Adding route for prefix 2600:1900:4140:4163:0:a::/128
2024-05-08T09:59:04+02:00 DEBG client/internal/routemanager/systemops.go:87: Route for 2600:1900:4140:4163:0:a::: interface &{28 1380 utun8  up|pointtopoint|multicast|running}, nexthop fe80::, preferred source fe80::5411:46d9:db4e:a562
2024-05-08T09:59:04+02:00 DEBG client/internal/routemanager/systemops.go:180: Found next hop fe80::%utun8 for prefix 2600:1900:4140:4163:0:a::/128 with interface &{28 1380 utun8  up|pointtopoint|multicast|running}
2024-05-08T09:59:04+02:00 DEBG client/internal/routemanager/systemops.go:201: Adding a new route for prefix 2600:1900:4140:4163:0:a::/128 with next hop fe80::%utun8
2024-05-08T09:59:04+02:00 DEBG client/internal/routemanager/routemanager.go:51: Increasing route ref count 1 for prefix 34.118.77.27/32
2024-05-08T09:59:04+02:00 DEBG client/internal/routemanager/routemanager.go:51: Increasing route ref count 1 for prefix 2600:1900:4140:4163:0:a::/128
2024-05-08T09:59:04+02:00 INFO client/internal/routemanager/manager.go:88: Routing setup complete
2024-05-08T09:59:04+02:00 DEBG iface/wg_configurer_usp.go:37: adding Wireguard private key
2024-05-08T09:59:04+02:00 DEBG iface/tun_darwin.go:83: device is ready to use: utun100
2024-05-08T09:59:04+02:00 DEBG signal/client/grpc.go:137: signal connection state READY
2024-05-08T09:59:04+02:00 INFO signal/client/grpc.go:158: connected to the Signal Service stream
2024-05-08T09:59:04+02:00 DEBG client/internal/engine.go:596: connecting to Management Service updates stream
2024-05-08T09:59:04+02:00 DEBG management/client/grpc.go:120: management connection state READY
2024-05-08T09:59:04+02:00 INFO client/internal/connect.go:247: Netbird engine started, my IP is: 100.64.240.231/16
2024-05-08T09:59:04+02:00 INFO management/client/grpc.go:147: connected to the Management Service stream
2024-05-08T09:59:04+02:00 DEBG management/client/grpc.go:253: got an update message from Management Service
2024-05-08T09:59:04+02:00 DEBG client/internal/engine.go:622: got TURNs update from Management Service, updating
2024-05-08T09:59:04+02:00 DEBG client/internal/engine.go:604: got STUNs update from Management Service, updating
2024-05-08T09:59:04+02:00 WARN client/internal/engine.go:507: running SSH server is not permitted
2024-05-08T09:59:04+02:00 DEBG client/internal/engine.go:653: got peers update from Management Service, total peers to connect to = 1
2024-05-08T09:59:04+02:00 DEBG client/internal/engine.go:885: creating peer connection KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=
2024-05-08T09:59:04+02:00 DEBG client/internal/routemanager/client.go:332: Received a new client network route update for 34.118.116.195/32
2024-05-08T09:59:04+02:00 DEBG client/internal/routemanager/client.go:332: Received a new client network route update for 34.118.106.37/32
2024-05-08T09:59:04+02:00 WARN client/internal/routemanager/client.go:153: the network 34.118.116.195/32 has not been assigned a routing peer as no peers from the list [KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=] are currently connected
2024-05-08T09:59:04+02:00 WARN client/internal/routemanager/client.go:153: the network 34.118.106.37/32 has not been assigned a routing peer as no peers from the list [KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=] are currently connected
2024-05-08T09:59:04+02:00 DEBG client/internal/routemanager/client.go:332: Received a new client network route update for 34.116.162.131/32
2024-05-08T09:59:04+02:00 WARN client/internal/routemanager/client.go:153: the network 34.116.162.131/32 has not been assigned a routing peer as no peers from the list [KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=] are currently connected
2024-05-08T09:59:04+02:00 DEBG client/internal/routemanager/client.go:332: Received a new client network route update for 34.116.146.33/32
2024-05-08T09:59:04+02:00 WARN client/internal/routemanager/client.go:153: the network 34.116.146.33/32 has not been assigned a routing peer as no peers from the list [KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=] are currently connected
2024-05-08T09:59:04+02:00 DEBG client/internal/routemanager/client.go:332: Received a new client network route update for 34.117.118.44/32
2024-05-08T09:59:04+02:00 WARN client/internal/routemanager/client.go:153: the network 34.117.118.44/32 has not been assigned a routing peer as no peers from the list [KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=] are currently connected
2024-05-08T09:59:04+02:00 DEBG client/internal/dns/service_memory.go:51: dns service listening on: 100.64.255.254
2024-05-08T09:59:04+02:00 INFO client/internal/dns/host_darwin.go:91: removing match domains from the system
2024-05-08T09:59:04+02:00 INFO client/internal/dns/host_darwin.go:157: added 1 search domains to the state. Domain list: netbird.selfhosted
2024-05-08T09:59:04+02:00 INFO client/internal/acl/manager.go:52: ACL rules processed in: 98.5µs, total rules count: 2
2024-05-08T09:59:06+02:00 DEBG client/internal/peer/conn.go:280: trying to connect to peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=
2024-05-08T09:59:06+02:00 DEBG client/internal/peer/conn.go:312: connection offer sent to peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=, waiting for the confirmation
2024-05-08T09:59:06+02:00 DEBG client/internal/peer/conn.go:763: OnRemoteAnswer from peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4= on status Disconnected
2024-05-08T09:59:06+02:00 DEBG client/internal/peer/conn.go:333: received connection confirmation from peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4= running version 0.27.3 and with remote WireGuard listen port 51820
2024-05-08T09:59:06+02:00 DEBG client/internal/peer/conn.go:661: peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4= ICE ConnectionState has changed to Checking
2024-05-08T09:59:06+02:00 DEBG client/internal/peer/conn.go:620: discovered local candidate udp4 host 192.168.33.3:51820
2024-05-08T09:59:06+02:00 DEBG client/internal/peer/conn.go:620: discovered local candidate udp4 host 192.168.64.1:51820
2024-05-08T09:59:06+02:00 DEBG client/internal/peer/conn.go:777: OnRemoteCandidate from peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4= -> udp4 host 172.17.0.2:51820
2024-05-08T09:59:06+02:00 DEBG iface/bind/udp_mux.go:346: ICE: registered 172.17.0.2:51820 for hzHrdmuiLYtXlaVb
2024-05-08T09:59:06+02:00 DEBG util/net/listener_generic.go:120: Listener resolved IP for 34.118.77.27:3478: 34.118.77.27
2024-05-08T09:59:06+02:00 DEBG client/internal/routemanager/routemanager.go:51: Increasing route ref count 2 for prefix 34.118.77.27/32
2024-05-08T09:59:06+02:00 DEBG client/internal/peer/conn.go:777: OnRemoteCandidate from peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4= -> udp4 srflx 34.118.77.27:51820 related 0.0.0.0:51820
2024-05-08T09:59:06+02:00 DEBG iface/bind/udp_mux.go:346: ICE: registered 34.118.77.27:51820 for hzHrdmuiLYtXlaVb
2024-05-08T09:59:06+02:00 DEBG client/internal/peer/conn.go:777: OnRemoteCandidate from peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4= -> udp4 relay 34.118.77.27:64341 related 0.0.0.0:48510
2024-05-08T09:59:06+02:00 DEBG iface/bind/udp_mux.go:346: ICE: registered 34.118.77.27:64341 for hzHrdmuiLYtXlaVb
2024-05-08T09:59:06+02:00 DEBG iface/bind/udp_mux.go:346: ICE: registered 172.17.0.2:51820 for hzHrdmuiLYtXlaVbstun:netbird.iceolab.com:3478
2024-05-08T09:59:06+02:00 DEBG iface/bind/udp_mux.go:346: ICE: registered 34.118.77.27:51820 for hzHrdmuiLYtXlaVbstun:netbird.iceolab.com:3478
2024-05-08T09:59:06+02:00 DEBG iface/bind/udp_mux.go:346: ICE: registered 34.118.77.27:64341 for hzHrdmuiLYtXlaVbstun:netbird.iceolab.com:3478
2024-05-08T09:59:06+02:00 DEBG client/internal/peer/conn.go:620: discovered local candidate udp4 srflx 193.25.251.165:51820 related 0.0.0.0:51820
2024-05-08T09:59:06+02:00 DEBG client/internal/peer/conn.go:620: discovered local candidate udp4 relay 34.118.77.27:61511 related 0.0.0.0:59550
2024-05-08T09:59:07+02:00 DEBG client/internal/peer/conn.go:425: setup relay connection
2024-05-08T09:59:07+02:00 DEBG client/internal/wgproxy/proxy_userspace.go:25: instantiate new userspace proxy
2024-05-08T09:59:07+02:00 ERRO util/net/dialer_generic.go:64: Failed to call dialer hooks: failed to resolve address :51820: lookup : no such host
2024-05-08T09:59:07+02:00 DEBG client/internal/peer/conn.go:655: selected candidate pair [local <-> remote] -> [udp4 relay 34.118.77.27:61511 related 0.0.0.0:59550 <-> udp4 host 172.17.0.2:51820], peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=
2024-05-08T09:59:07+02:00 DEBG client/internal/peer/conn.go:661: peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4= ICE ConnectionState has changed to Connected
2024-05-08T09:59:07+02:00 DEBG client/internal/peer/conn.go:439: Conn resolved IP for 127.0.0.1:49469: 127.0.0.1
2024-05-08T09:59:07+02:00 DEBG client/internal/routemanager/routemanager.go:51: Increasing route ref count 0 for prefix 127.0.0.1/32
2024-05-08T09:59:07+02:00 DEBG client/internal/routemanager/routemanager.go:55: Adding route for prefix 127.0.0.1/32
2024-05-08T09:59:07+02:00 DEBG client/internal/routemanager/routemanager.go:61: Adding route for prefix 127.0.0.1/32: route not allowed
2024-05-08T09:59:07+02:00 ERRO client/internal/peer/conn.go:444: Before add peer hook failed: adding route reference: failed to add route for prefix 127.0.0.1/32: route not allowed
2024-05-08T09:59:07+02:00 DEBG iface/iface.go:79: updating interface utun100 peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=, endpoint 127.0.0.1:49469
2024-05-08T09:59:07+02:00 INFO client/internal/peer/conn.go:388: connected to peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=, endpoint address: 127.0.0.1:49469
2024-05-08T09:59:07+02:00 DEBG client/internal/routemanager/client.go:178: triggered route state update for Peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=, state: Connected
2024-05-08T09:59:07+02:00 INFO client/internal/routemanager/client.go:158: new chosen route is cokalj9mjugs73f8kfbg with peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4= with score 0.985499 for network 34.118.116.195/32
2024-05-08T09:59:07+02:00 INFO client/internal/routemanager/client.go:158: new chosen route is coof3gpmjugs73f8kgig with peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4= with score 99980.985499 for network 34.117.118.44/32
2024-05-08T09:59:07+02:00 DEBG client/internal/routemanager/client.go:178: triggered route state update for Peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=, state: Connected
2024-05-08T09:59:07+02:00 INFO client/internal/routemanager/client.go:158: new chosen route is cojqhopmjugs73f8kf80 with peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4= with score 0.985499 for network 34.118.106.37/32
2024-05-08T09:59:07+02:00 DEBG client/internal/routemanager/client.go:178: triggered route state update for Peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=, state: Connected
2024-05-08T09:59:07+02:00 DEBG client/internal/routemanager/client.go:178: triggered route state update for Peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=, state: Connected
2024-05-08T09:59:07+02:00 INFO client/internal/routemanager/client.go:158: new chosen route is cokal7pmjugs73f8kfb0 with peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4= with score 0.985499 for network 34.116.162.131/32
2024-05-08T09:59:07+02:00 WARN client/internal/routemanager/systemops.go:259: Skipping adding a new route for network 34.116.162.131/32 because it already exists
2024-05-08T09:59:07+02:00 DEBG iface/iface.go:97: adding allowed IP to interface utun100 and peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=: allowed IP 34.116.162.131/32
2024-05-08T09:59:07+02:00 DEBG client/internal/routemanager/client.go:178: triggered route state update for Peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=, state: Connected
2024-05-08T09:59:07+02:00 INFO client/internal/routemanager/client.go:158: new chosen route is cojqhj1mjugs73f8kf7g with peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4= with score 0.985499 for network 34.116.146.33/32
2024-05-08T09:59:07+02:00 DEBG client/internal/routemanager/client.go:178: triggered route state update for Peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=, state: Connected
2024-05-08T09:59:07+02:00 WARN client/internal/routemanager/systemops.go:259: Skipping adding a new route for network 34.116.146.33/32 because it already exists
2024-05-08T09:59:07+02:00 DEBG iface/iface.go:97: adding allowed IP to interface utun100 and peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=: allowed IP 34.116.146.33/32
2024-05-08T09:59:07+02:00 DEBG client/internal/routemanager/client.go:178: triggered route state update for Peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=, state: Connected
2024-05-08T09:59:07+02:00 DEBG client/internal/routemanager/client.go:178: triggered route state update for Peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=, state: Connected
2024-05-08T09:59:07+02:00 DEBG iface/iface.go:97: adding allowed IP to interface utun100 and peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=: allowed IP 34.117.118.44/32
2024-05-08T09:59:07+02:00 DEBG client/internal/routemanager/client.go:178: triggered route state update for Peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=, state: Connected
2024-05-08T09:59:07+02:00 DEBG client/internal/routemanager/client.go:178: triggered route state update for Peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=, state: Connected
2024-05-08T09:59:07+02:00 DEBG client/internal/routemanager/client.go:178: triggered route state update for Peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=, state: Connected
2024-05-08T09:59:07+02:00 DEBG iface/iface.go:97: adding allowed IP to interface utun100 and peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=: allowed IP 34.118.106.37/32
2024-05-08T09:59:07+02:00 DEBG client/internal/routemanager/client.go:178: triggered route state update for Peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=, state: Connected
2024-05-08T09:59:07+02:00 DEBG client/internal/routemanager/client.go:178: triggered route state update for Peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=, state: Connected
2024-05-08T09:59:07+02:00 DEBG iface/iface.go:97: adding allowed IP to interface utun100 and peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=: allowed IP 34.118.116.195/32
2024-05-08T09:59:07+02:00 DEBG client/internal/routemanager/client.go:178: triggered route state update for Peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=, state: Connected
2024-05-08T09:59:07+02:00 DEBG client/internal/routemanager/client.go:178: triggered route state update for Peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=, state: Connected
2024-05-08T09:59:07+02:00 DEBG client/internal/routemanager/client.go:178: triggered route state update for Peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=, state: Connected
2024-05-08T09:59:07+02:00 DEBG client/internal/routemanager/client.go:178: triggered route state update for Peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=, state: Connected

--- ROUTES

$ netstat -rn | grep '34\.'
34.117.118.44      utun100            UHS               utun100
34.118.77.27       192.168.33.1       UGHS                  en0
34.118.106.37      utun100            UHS               utun100
34.118.116.195     utun100            UHS               utun100

$ netstat -nra | grep '34\.'
34.116.146.33      192.168.33.1       UGHWIig               en0
34.116.162.131     192.168.33.1       UGHWIig               en0
34.117.118.44      utun100            UHS               utun100
34.118.77.27       192.168.33.1       UGHS                  en0
34.118.77.27       192.168.33.1       UGHWIig               en0
34.118.106.37      utun100            UHS               utun100
34.118.116.195     utun100            UHS               utun100

--- DISCONNECT

2024-05-08T10:00:01+02:00 DEBG client/internal/engine.go:384: removing all peer connections
2024-05-08T10:00:01+02:00 DEBG client/internal/engine.go:396: removing peer from engine KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=
2024-05-08T10:00:01+02:00 DEBG client/internal/peer/conn.go:517: trying to cleanup KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=
2024-05-08T10:00:01+02:00 DEBG client/internal/routemanager/client.go:315: stopping watcher for network 34.118.106.37/32
2024-05-08T10:00:01+02:00 DEBG client/internal/routemanager/client.go:315: stopping watcher for network 34.118.116.195/32
2024-05-08T10:00:01+02:00 DEBG client/internal/routemanager/client.go:315: stopping watcher for network 34.116.146.33/32
2024-05-08T10:00:01+02:00 DEBG client/internal/routemanager/client.go:315: stopping watcher for network 34.117.118.44/32
2024-05-08T10:00:01+02:00 DEBG client/internal/routemanager/client.go:315: stopping watcher for network 34.116.162.131/32
2024-05-08T10:00:01+02:00 DEBG signal/client/grpc.go:389: stream canceled (usually indicates shutdown)
2024-05-08T10:00:01+02:00 DEBG signal/client/grpc.go:164: signal connection context has been canceled, this usually indicates shutdown
2024-05-08T10:00:01+02:00 DEBG management/client/grpc.go:249: disconnected from Management Service sync stream: rpc error: code = Canceled desc = context canceled
2024-05-08T10:00:01+02:00 DEBG management/client/grpc.go:157: management connection context has been canceled, this usually indicates shutdown
2024-05-08T10:00:01+02:00 DEBG client/internal/engine.go:594: stopped receiving updates from Management Service
2024-05-08T10:00:01+02:00 DEBG client/internal/routemanager/routemanager.go:90: Decreasing route ref count 3 for prefix 34.118.77.27/32
2024-05-08T10:00:01+02:00 DEBG client/internal/routemanager/routemanager.go:83: No prefixes found for connection ID 26363ac1-6da1-47b5-b1b0-372c4ce3e87d
2024-05-08T10:00:01+02:00 DEBG iface/iface.go:88: Removing peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4= from interface utun100
2024-05-08T10:00:01+02:00 DEBG client/internal/peer/conn.go:661: peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4= ICE ConnectionState has changed to Closed
2024-05-08T10:00:01+02:00 DEBG client/internal/routemanager/routemanager.go:83: No prefixes found for connection ID c5cc140d-17de-493e-8034-d74ad9faece9
2024-05-08T10:00:02+02:00 DEBG client/internal/peer/conn.go:569: error while updating peer's KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4= state, err: peer doesn't exist
2024-05-08T10:00:02+02:00 DEBG client/internal/peer/conn.go:572: failed to reset wireguard stats for peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=: peer doesn't exist
2024-05-08T10:00:02+02:00 DEBG client/internal/peer/conn.go:575: cleaned up connection to peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4=
2024-05-08T10:00:02+02:00 DEBG client/internal/engine.go:866: connection to peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4= failed: connection to peer KBvYvOCaHHd6LOlr/nf6njlILsi9M2Bxkq8daIoiDF4= has been closed
2024-05-08T10:00:02+02:00 ERRO client/internal/routemanager/client.go:318: Couldn't remove route from peer and system for network 34.116.146.33/32: remove route: get peer state: peer not found
2024-05-08T10:00:02+02:00 ERRO client/internal/routemanager/client.go:318: Couldn't remove route from peer and system for network 34.117.118.44/32: remove route: get peer state: peer not found
2024-05-08T10:00:02+02:00 ERRO client/internal/routemanager/client.go:318: Couldn't remove route from peer and system for network 34.116.162.131/32: remove route: get peer state: peer not found
2024-05-08T10:00:02+02:00 ERRO client/internal/routemanager/client.go:318: Couldn't remove route from peer and system for network 34.118.116.195/32: remove route: get peer state: peer not found
2024-05-08T10:00:02+02:00 ERRO client/internal/routemanager/client.go:318: Couldn't remove route from peer and system for network 34.118.106.37/32: remove route: get peer state: peer not found
2024-05-08T10:00:02+02:00 INFO client/internal/dns/host_darwin.go:120: removing search domains from system
2024-05-08T10:00:02+02:00 DEBG client/internal/routemanager/routemanager.go:116: Removing route for prefix 34.118.77.27/32
2024-05-08T10:00:02+02:00 DEBG client/internal/routemanager/routemanager.go:116: Removing route for prefix 2600:1900:4140:4163:0:a::/128
2024-05-08T10:00:02+02:00 INFO client/internal/routemanager/manager.go:112: Routing cleanup complete
2024-05-08T10:00:02+02:00 DEBG client/internal/engine.go:1127: removing Netbird interface utun100
2024-05-08T10:00:02+02:00 INFO client/internal/engine.go:224: stopped Netbird Engine
2024-05-08T10:00:02+02:00 INFO client/internal/connect.go:261: stopped NetBird client

--- ROUTES

$ netstat -rn | grep '34\.'

$ netstat -nra | grep '34\.'
34.116.146.33      192.168.33.1       UGHWIig               en0
34.116.162.131     192.168.33.1       UGHWIig               en0
34.118.77.27       192.168.33.1       UGHWIig               en0

The bug is reproducible, but IP addresses differs. There is no such case that e.g. 34.118.77.27 is always there. Sometime it works sometime another route works 😅