search

netbirdio / netbird

Connect your devices into a secure WireGuard®-based overlay network with SSO, MFA and granular access controls.
https://netbird.io
BSD 3-Clause "New" or "Revised" License
10.73k stars 483 forks source link

Client does not create host route to the exit nodes / peers #2421

Closed Takalele closed 1 month ago

Takalele commented 1 month ago

Describe the problem

When a Exit Node is defined and enabled, a client with that "default route" enabled will send ALL traffic through the tunnel creating a blackhole/loop.

adding the hostroute manually fixed solves the problem.

To Reproduce

Steps to reproduce the behavior:

  1. Add Exit Node
  2. Enabled it on the client
  3. backhole created
  4. See error

Expected behavior

the client creates a hostroute (/32) to each peer based of the original default route.

Are you using NetBird Cloud?

Please specify whether you use NetBird Cloud or self-host NetBird's control plane.

NetBird selfhosted

netbird 0.28.7

NetBird status -dA output:

C:\Windows\System32>netbird status -dA
Peers detail:
 exit-node-vpn.netbird.selfhosted:
  NetBird IP: 100.105.9.17
  Public key: SuE8Hl0ChqsnMfR2/TrxFBtJrJV5ZbOV2dcnPitnBkA=
  Status: Connected
  -- detail --
  Connection type: P2P
  Direct: true
  ICE candidate (Local/Remote): host/host
  ICE candidate endpoints (Local/Remote): 192.168.48.1:51820/198.51.100.0:51820
  Last connection update: 23 minutes, 45 seconds ago
  Last WireGuard handshake: 1 minute, 42 seconds ago
  Transfer status (received/sent) 12.3 MiB/8.1 MiB
  Quantum resistance: false
  Routes: 0.0.0.0/0
  Latency: 20.6718ms

 handy.netbird.selfhosted:
  NetBird IP: 100.105.33.250
  Public key: KqK8OQ6jSinnSU/yBYNCS35GKRj5FLHWlffDkeXcfT8=
  Status: Connected
  -- detail --
  Connection type: Relayed
  Direct: false
  ICE candidate (Local/Remote): relay/prflx
  ICE candidate endpoints (Local/Remote): 198.51.100.0:62997/198.51.100.1:25439
  Last connection update: 8 minutes, 30 seconds ago
  Last WireGuard handshake: 2 seconds ago
  Transfer status (received/sent) 1.6 KiB/1.3 KiB
  Quantum resistance: false
  Routes: -
  Latency: 171.03ms

 dsda-1.netbird.selfhosted:
  NetBird IP: 100.105.191.186
  Public key: w7Bd10GqBG98bM+2oqJmNlDvXRzjfmFHiv5AGhfm6CU=
  Status: Connected
  -- detail --
  Connection type: Relayed
  Direct: false
  ICE candidate (Local/Remote): relay/srflx
  ICE candidate endpoints (Local/Remote): 198.51.100.0:58185/198.51.100.2:51820
  Last connection update: 23 minutes, 40 seconds ago
  Last WireGuard handshake: 44 seconds ago
  Transfer status (received/sent) 3.8 KiB/2.8 KiB
  Quantum resistance: false
  Routes: -
  Latency: 37.3066ms

 is2vbb5.netbird.selfhosted:
  NetBird IP: 100.105.230.218
  Public key: KxNpJNcvJ4yckKzvqkAo4G4AB2o1zJLWr6zg04lpsFs=
  Status: Disconnected
  -- detail --
  Connection type: Relayed
  Direct: false
  ICE candidate (Local/Remote): relay/prflx
  ICE candidate endpoints (Local/Remote): 198.51.100.0:62997/198.51.100.1:25439
  Last connection update: -
  Last WireGuard handshake: 2 seconds ago
  Transfer status (received/sent) 1.6 KiB/1.3 KiB
  Quantum resistance: false
  Routes: -
  Latency: 0s

OS: windows/amd64
Daemon version: 0.28.7
CLI version: 0.28.7
Management: Connected to https://vpn.anon-nDrWh.domain:443
Signal: Connected to https://vpn.anon-nDrWh.domain:443
Relays:
  [stun:turn.anon-nDrWh.domain:3478] is Available
  [turn:turn.anon-nDrWh.domain:3478?transport=udp] is Available
Nameservers:
FQDN: voyager-1.netbird.selfhosted
NetBird IP: 100.105.209.49/16
Interface type: Userspace
Quantum resistance: false
Routes: -
Peers count: 3/4 Connected
pascal-fischer commented 1 month ago

Hi @Takalele, I have some trouble understanding what could be going wrong here. Can you share some logs before adding any routes to the peer (following https://docs.netbird.io/how-to/troubleshooting-client#debug-bundle). And a snapshot of the important parts of the routing table before, with netbird and after you added the route? (maybe in some redacted format)

Takalele commented 1 month ago

hi, @pascal-fischer

Unfortunately, I’m unable to address that at the moment as I’m encountering another issue. While the login is successful, the client is throwing the following error: image mgmt logs:

WARN [context: GRPC, requestID: 6f5d8624-f48f-4a80-9118-70e385cdbd9b, accountID: UNKNOWN, peerID: xyz] management/server/grpcserver.go:456: failed validating JWT token sent
from peer xyz with error rpc error: code = InvalidArgument desc = invalid jwt token, err: Error parsing token: Token used before issued. Trying again as it may be due to the IdP cache issue

The same issue also occurs when I log into the admin portal, but it resolves itself after a couple of seconds

2024-08-13T17:07:53Z ERRO [context: HTTP, requestID: 3ac89db3-ea21-4957-afa9-104c0a390af7] management/server/jwtclaims/jwtValidator.go:161: error parsing token: Token used before issued
2024-08-13T17:07:53Z ERRO [context: HTTP, requestID: 3ac89db3-ea21-4957-afa9-104c0a390af7] management/server/http/middleware/auth_middleware.go:89: Error when validating JWT claims: Error parsing token: Token used before issued
2024-08-13T17:07:53Z ERRO [context: HTTP, requestID: 3ac89db3-ea21-4957-afa9-104c0a390af7] management/server/http/util/util.go:81: got a handler error: token invalid
2024-08-13T17:07:53Z ERRO [context: HTTP, requestID: 3ac89db3-ea21-4957-afa9-104c0a390af7] management/server/telemetry/http_api_metrics.go:191: HTTP response 3ac89db3-ea21-4957-afa9-104c0a390af7: GET /api/users status 401
2024-08-13T17:07:54Z ERRO [context: HTTP, requestID: f4b5bd0b-ed84-400f-9581-853ab5f1efb4] management/server/jwtclaims/jwtValidator.go:161: error parsing token: Token used before issued
2024-08-13T17:07:54Z ERRO [context: HTTP, requestID: f4b5bd0b-ed84-400f-9581-853ab5f1efb4] management/server/http/middleware/auth_middleware.go:89: Error when validating JWT claims: Error parsing token: Token used before issued
2024-08-13T17:07:54Z ERRO [context: HTTP, requestID: f4b5bd0b-ed84-400f-9581-853ab5f1efb4] management/server/http/util/util.go:81: got a handler error: token invalid
2024-08-13T17:07:54Z ERRO [context: HTTP, requestID: f4b5bd0b-ed84-400f-9581-853ab5f1efb4] management/server/telemetry/http_api_metrics.go:191: HTTP response f4b5bd0b-ed84-400f-9581-853ab5f1efb4: GET /api/users status 401
2024-08-13T17:07:54Z ERRO [requestID: c7244da2-af01-4289-9c40-301a95847b23, context: HTTP] management/server/jwtclaims/jwtValidator.go:161: error parsing token: Token used before issued
2024-08-13T17:07:54Z ERRO [context: HTTP, requestID: c7244da2-af01-4289-9c40-301a95847b23] management/server/http/middleware/auth_middleware.go:89: Error when validating JWT claims: Error parsing token: Token used before issued
2024-08-13T17:07:54Z ERRO [context: HTTP, requestID: c7244da2-af01-4289-9c40-301a95847b23] management/server/http/util/util.go:81: got a handler error: token invalid
2024-08-13T17:07:54Z ERRO [context: HTTP, requestID: c7244da2-af01-4289-9c40-301a95847b23] management/server/telemetry/http_api_metrics.go:191: HTTP response c7244da2-af01-4289-9c40-301a95847b23: GET /api/users status 401
2024-08-13T17:07:55Z ERRO [context: HTTP, requestID: 93721015-4a45-4b48-96e2-425308036673] management/server/jwtclaims/jwtValidator.go:161: error parsing token: Token used before issued
2024-08-13T17:07:55Z ERRO [requestID: 93721015-4a45-4b48-96e2-425308036673, context: HTTP] management/server/http/middleware/auth_middleware.go:89: Error when validating JWT claims: Error parsing token: Token used before issued
2024-08-13T17:07:55Z ERRO [context: HTTP, requestID: 93721015-4a45-4b48-96e2-425308036673] management/server/http/util/util.go:81: got a handler error: token invalid
2024-08-13T17:07:55Z ERRO [context: HTTP, requestID: 93721015-4a45-4b48-96e2-425308036673] management/server/telemetry/http_api_metrics.go:191: HTTP response 93721015-4a45-4b48-96e2-425308036673: GET /api/users status 401

I suspected a clock issue, but both systems and all the containers have the correct time.

any ideas?

pascal-fischer commented 1 month ago

What is the IDP that you are using? You need to make sure the time used for the container running your IDP and the time used where the management container is running are in sync, down to the second. You can use sudo ntpdate pool.ntp.org to sync the time of those machines. Afterwards you need to restart the services.

Takalele commented 1 month ago

@pascal-fischer it was definitely a time synchronization issue. The host running Netbird was delayed by 2 seconds. Unfortunately, I don't have control over that host, so I had to migrate the entire setup to another cluster.

However, I'm now ready for debugging. The first thing I had to do was disable the network monitor due to Issue 2395

Client nat IP: eg .:10.8.2.36 Client Real-IP: eg.: 212.1.1.2

Peer/Exit Node Real-IP: 198.51.100.1

Client routing table: routing table before the connection:

Aktive Routen:
     Netzwerkziel    Netzwerkmaske          Gateway    Schnittstelle Metrik
          0.0.0.0          0.0.0.0         10.8.2.1        10.8.2.36     40
         10.8.2.0    255.255.255.0   Auf Verbindung         10.8.2.36    296
        10.8.2.36  255.255.255.255   Auf Verbindung         10.8.2.36    296
       10.8.2.255  255.255.255.255   Auf Verbindung         10.8.2.36    296
        127.0.0.0        255.0.0.0   Auf Verbindung         127.0.0.1    331
        127.0.0.1  255.255.255.255   Auf Verbindung         127.0.0.1    331
  127.255.255.255  255.255.255.255   Auf Verbindung         127.0.0.1    331
     172.22.240.0    255.255.240.0   Auf Verbindung      172.22.240.1   5256
     172.22.240.1  255.255.255.255   Auf Verbindung      172.22.240.1   5256
   172.22.255.255  255.255.255.255   Auf Verbindung      172.22.240.1   5256
     172.29.160.0    255.255.240.0   Auf Verbindung      172.29.160.1   5256
     172.29.160.1  255.255.255.255   Auf Verbindung      172.29.160.1   5256
   172.29.175.255  255.255.255.255   Auf Verbindung      172.29.160.1   5256
        224.0.0.0        240.0.0.0   Auf Verbindung         127.0.0.1    331
        224.0.0.0        240.0.0.0   Auf Verbindung         10.8.2.36    296
        224.0.0.0        240.0.0.0   Auf Verbindung      172.29.160.1   5256
        224.0.0.0        240.0.0.0   Auf Verbindung      172.22.240.1   5256
  255.255.255.255  255.255.255.255   Auf Verbindung         127.0.0.1    331
  255.255.255.255  255.255.255.255   Auf Verbindung         10.8.2.36    296
  255.255.255.255  255.255.255.255   Auf Verbindung      172.29.160.1   5256
  255.255.255.255  255.255.255.255   Auf Verbindung      172.22.240.1   5256

routing table during the connection:

Aktive Routen:
     Netzwerkziel    Netzwerkmaske          Gateway    Schnittstelle Metrik
          0.0.0.0          0.0.0.0         10.8.2.1        10.8.2.36     40
          0.0.0.0        128.0.0.0   Auf Verbindung   100.105.102.132      6
         10.8.2.0    255.255.255.0   Auf Verbindung         10.8.2.36    296
        10.8.2.36  255.255.255.255   Auf Verbindung         10.8.2.36    296
       10.8.2.255  255.255.255.255   Auf Verbindung         10.8.2.36    296
      100.105.0.0      255.255.0.0   Auf Verbindung   100.105.102.132    261
  100.105.102.132  255.255.255.255   Auf Verbindung   100.105.102.132    261
  100.105.255.255  255.255.255.255   Auf Verbindung   100.105.102.132    261
        127.0.0.0        255.0.0.0   Auf Verbindung         127.0.0.1    331
        127.0.0.1  255.255.255.255   Auf Verbindung         127.0.0.1    331
  127.255.255.255  255.255.255.255   Auf Verbindung         127.0.0.1    331
  127.255.255.255  255.255.255.255   Auf Verbindung   100.105.102.132    261
        128.0.0.0        128.0.0.0   Auf Verbindung   100.105.102.132      6
     172.22.240.0    255.255.240.0   Auf Verbindung      172.22.240.1   5256
     172.22.240.1  255.255.255.255   Auf Verbindung      172.22.240.1   5256
   172.22.255.255  255.255.255.255   Auf Verbindung      172.22.240.1   5256
     172.29.160.0    255.255.240.0   Auf Verbindung      172.29.160.1   5256
     172.29.160.1  255.255.255.255   Auf Verbindung      172.29.160.1   5256
   172.29.175.255  255.255.255.255   Auf Verbindung      172.29.160.1   5256
     192.168.48.0    255.255.240.0   Auf Verbindung      192.168.48.1   5256
     192.168.48.1  255.255.255.255   Auf Verbindung      192.168.48.1   5256
   192.168.63.255  255.255.255.255   Auf Verbindung      192.168.48.1   5256
        224.0.0.0        240.0.0.0   Auf Verbindung         127.0.0.1    331
        224.0.0.0        240.0.0.0   Auf Verbindung         10.8.2.36    296
        224.0.0.0        240.0.0.0   Auf Verbindung      172.29.160.1   5256
        224.0.0.0        240.0.0.0   Auf Verbindung      172.22.240.1   5256
        224.0.0.0        240.0.0.0   Auf Verbindung      192.168.48.1   5256
        224.0.0.0        240.0.0.0   Auf Verbindung   100.105.102.132    261
  255.255.255.255  255.255.255.255   Auf Verbindung         127.0.0.1    331
  255.255.255.255  255.255.255.255   Auf Verbindung         10.8.2.36    296
  255.255.255.255  255.255.255.255   Auf Verbindung      172.29.160.1   5256
  255.255.255.255  255.255.255.255   Auf Verbindung      172.22.240.1   5256
  255.255.255.255  255.255.255.255   Auf Verbindung      192.168.48.1   5256
  255.255.255.255  255.255.255.255   Auf Verbindung   100.105.102.132    261

Peer/Exit Node routing table: routing table before the connection and during the connection:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         0.0.0.0         0.0.0.0         U     0      0        0 venet0
10.255.254.0    0.0.0.0         255.255.255.0   U     0      0        0 tap_soft-vpn
100.105.0.0     0.0.0.0         255.255.0.0     U     0      0        0 wt0
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
172.18.0.0      0.0.0.0         255.255.255.0   U     0      0        0 proxy
198.51.100.0    0.0.0.0         255.255.255.0   U     0      0        0 venet0

During testing, I noticed that when the connection was terminated, the manually added route was automatically deleted.

BR Takalele

netbird.debug.1315957774.zip

lixmal commented 1 month ago

Hi @Takalele, for some reason your debug bundle does not contain trace/debug level logs. Could you run netbird debug log level trace and then run netbird debug for 2m -AS again, please?

Additionally, to help resolve the network monitor issue, could you provide Get-NetAdapter (from powershell), please? We're interested in whether the virtual interface(s) share mac addresses with other interfaces so we can exclude them from the availability check.

Takalele commented 1 month ago

Hi @lixmal, sure:

Microsoft Windows [Version 10.0.22631.3958]
(c) Microsoft Corporation. Alle Rechte vorbehalten.

C:\Windows\System32>netbird debug log level trace
Log level set successfully to trace

C:\Windows\System32>netbird up --network-monitor=false
Connected

C:\Windows\System32>netbird debug for 2m -AS
Netbird down
Netbird up
Remaining time: 00:00:01
Duration completed
Creating debug bundle...
C:\WINDOWS\SystemTemp\netbird.debug.3155538032.zip

C:\Windows\System32>netbird up --network-monitor=true
Connected

C:\Windows\System32>netbird debug for 2m -AS
Netbird down
Netbird up
Remaining time: 00:00:01
Duration completed
Creating debug bundle...
C:\WINDOWS\SystemTemp\netbird.debug.1750120769.zip

Get-NetAdapter

PS C:\WINDOWS\system32> Get-NetAdapter

Name                      InterfaceDescription                    ifIndex Status       MacAddress             LinkSpeed
----                      --------------------                    ------- ------       ----------             ---------
Ethernet 2                Surface Ethernet Adapter                     30 Disconnected 1C-1A-DF-BF-73-7A          0 bps
WLAN                      Intel(R) Wi-Fi 6 AX200 160MHz                22 Up           6C-A1-00-53-DA-B2       300 Mbps
Bluetooth-Netzwerkverb... Bluetooth Device (Personal Area Netw...       9 Disconnected 6C-A1-00-53-DA-B6         3 Mbps
vEthernet (WLAN)          Hyper-V Virtual Ethernet Adapter #3           5 Disconnected 6C-A1-00-53-DA-B2        10 Gbps
VPN - VPN Client          VPN Client Adapter - VPN                      4 Disconnected 5E-87-00-E0-98-40       100 Mbps

logs:

netbird.debug.1750120769_network-monitor=true.zip netbird.debug.3155538032_network-monitor=false.zip

Takalele commented 1 month ago

@lixmal @pascal-fischer, The client must also add a host route for the TURN server; otherwise, it won't function properly if the TURN server is needed. This is particularly crucial in the case of an exit node or default route, as in such scenarios, the TURN server traffic will also be routed through the tunnel. To ensure that TURN works correctly, its traffic must not pass through the tunnel.

lixmal commented 1 month ago

Hi @Takalele,

thanks for the interfaces, that's helpful. Regarding the routes, there are the following logs

2024-08-15T14:07:43+02:00 DEBG iface/bind/udp_mux.go:346: ICE: registered 198.51.100.1:51820 for ddytZXiJzeKPzAhi
                                                                           ^-- our peer's IP

2024-08-15T14:07:43+02:00 DEBG util/net/listener_nonios.go:120: Listener resolved IP for 198.51.100.1:3478: 198.51.100.1
2024-08-15T14:07:43+02:00 TRAC client/internal/routemanager/refcounter/refcounter.go:54: Increasing ref count 0 for prefix 198.51.100.1/32 with [{invalid IP <nil>}]
2024-08-15T14:07:43+02:00 TRAC client/internal/routemanager/refcounter/refcounter.go:58: Adding for prefix 198.51.100.1/32 with [{invalid IP <nil>}]
2024-08-15T14:07:43+02:00 DEBG client/internal/routemanager/systemops/systemops_generic.go:167: Adding a new route for prefix 198.51.100.1/32 with next hop 10.8.2.1
2024-08-15T14:07:43+02:00 DEBG client/internal/peer/conn.go:603: discovered local candidate udp4 srflx 198.51.100.2:18212 related 0.0.0.0:51820
2024-08-15T14:07:43+02:00 DEBG client/internal/peer/conn.go:633: selected candidate pair [local <-> remote] -> [udp4 host 127.0.0.1:51820 <-> udp4 host 198.51.100.1:51820], peer KxNpJNcvJ4yckKzvqkAo4G4AB2o1zJLWr6zg04lpsFs=
2024-08-15T14:07:43+02:00 DEBG client/internal/peer/conn.go:639: peer KxNpJNcvJ4yckKzvqkAo4G4AB2o1zJLWr6zg04lpsFs= ICE ConnectionState has changed to Connected
2024-08-15T14:07:43+02:00 DEBG client/internal/peer/conn.go:412: Conn resolved IP for 198.51.100.1:51820: 198.51.100.1
2024-08-15T14:07:43+02:00 TRAC client/internal/routemanager/systemops/systemops_windows.go:212: route add 198.51.100.1/32 10.8.2.1 if 5:  OK!
                                                                                                                                          ^-- here the peer IP route was added, pointing to the default gw

2024-08-15T14:07:43+02:00 TRAC client/internal/routemanager/refcounter/refcounter.go:54: Increasing ref count 1 for prefix 198.51.100.1/32 with [{10.8.2.1 0xc000740540}]
2024-08-15T14:07:43+02:00 DEBG iface/iface.go:92: updating interface wt0 peer KxNpJNcvJ4yckKzvqkAo4G4AB2o1zJLWr6zg04lpsFs=, endpoint 198.51.100.1:51820
2024-08-15T14:07:43+02:00 INFO client/internal/peer/conn.go:362: connected to peer KxNpJNcvJ4yckKzvqkAo4G4AB2o1zJLWr6zg04lpsFs=, endpoint address: 198.51.100.1:51820

2024-08-15T14:07:43+02:00 INFO client/internal/routemanager/client.go:177: New chosen route is cqtt443iojjd2cor2ocg with peer KxNpJNcvJ4yckKzvqkAo4G4AB2o1zJLWr6zg04lpsFs= with score 2.000000 for network [0.0.0.0/0]
2024-08-15T14:07:43+02:00 TRAC client/internal/routemanager/refcounter/refcounter.go:54: Increasing ref count 0 for prefix 0.0.0.0/0 with [<nil>]
2024-08-15T14:07:43+02:00 TRAC client/internal/routemanager/refcounter/refcounter.go:58: Adding for prefix 0.0.0.0/0 with [<nil>]
2024-08-15T14:07:43+02:00 TRAC client/internal/routemanager/systemops/systemops_windows.go:212: route add 0.0.0.0/1 0.0.0.0 if 83:  OK!
2024-08-15T14:07:44+02:00 TRAC client/internal/routemanager/systemops/systemops_windows.go:212: route add 198.51.100.3/1 0.0.0.0 if 83:  OK!
2024-08-15T14:07:44+02:00 TRAC client/internal/routemanager/systemops/systemops_windows.go:212: route add ::/1 :: if 83:  OK!
2024-08-15T14:07:44+02:00 TRAC client/internal/routemanager/systemops/systemops_windows.go:212: route add 8000::/1 :: if 83:  OK!
                                                                                                                              ^-- here the default routes were added to the vpn interface

2024-08-15T14:07:44+02:00 TRAC client/internal/routemanager/refcounter/refcounter.go:54: Increasing ref count 0 for prefix 0.0.0.0/0 with []
2024-08-15T14:07:44+02:00 TRAC client/internal/routemanager/refcounter/refcounter.go:58: Adding for prefix 0.0.0.0/0 with []

So everything seems in order and I don't see any errors related to that. Is it possible something else keeps deleting these routes while netbird is active?

Takalele commented 1 month ago

Hi @lixmal,

I created a small PowerShell script to monitor the network routes (see below) in order to debug this issue further. You were right — there are routes being added, but they're being created on the wrong interface (ifIndex 80). In my case, the correct index would be 16. From the debugging, I discovered that this happens because, for some unknown Hyper-V reason, both interfaces have the same MAC address, even though no bridging is currently configured. When bridging is configured, there are three network interfaces with the same MAC address. In my tests, with bridging enabled, the routes were created correctly, but to prevent future issues, it could be good to use the ifIndex when creating the routes on windows systems.

After playing around with the Hyper-V virtual switch settings, i can no longer reproduce the issue, so it seems like this was a Hyper-V bug that has already been fixed.

Thanks for your help!

BR Takalele

Bad ifindex routes:

[2024-08-17 18:14:10] Route added: DestinationPrefix: 255.255.255.255/32, NextHop: 0.0.0.0, InterfaceIndex: 80, InterfaceAlias: wt0
[2024-08-17 18:14:10] Route added: DestinationPrefix: 224.0.0.0/4, NextHop: 0.0.0.0, InterfaceIndex: 80, InterfaceAlias: wt0
[2024-08-17 18:14:10] Route added: DestinationPrefix: 100.105.255.255/32, NextHop: 0.0.0.0, InterfaceIndex: 80, InterfaceAlias: wt0
[2024-08-17 18:14:10] Route added: DestinationPrefix: 100.105.102.132/32, NextHop: 0.0.0.0, InterfaceIndex: 80, InterfaceAlias: wt0
[2024-08-17 18:14:10] Route added: DestinationPrefix: 100.105.0.0/16, NextHop: 0.0.0.0, InterfaceIndex: 80, InterfaceAlias: wt0
[2024-08-17 18:14:10] Route added: DestinationPrefix: 198.51.100.3/32, NextHop: 10.8.2.1, InterfaceIndex: 5, InterfaceAlias: vEthernet (WLAN)
[2024-08-17 18:14:10] Route added: DestinationPrefix: ff00::/8, NextHop: ::, InterfaceIndex: 80, InterfaceAlias: wt0
[2024-08-17 18:14:16] Route added: DestinationPrefix: 198.51.100.1/32, NextHop: 10.8.2.1, InterfaceIndex: 5, InterfaceAlias: vEthernet (WLAN)
[2024-08-17 18:14:16] Route added: DestinationPrefix: 128.0.0.0/1, NextHop: 0.0.0.0, InterfaceIndex: 80, InterfaceAlias: wt0
[2024-08-17 18:14:16] Route added: DestinationPrefix: 127.255.255.255/32, NextHop: 0.0.0.0, InterfaceIndex: 80, InterfaceAlias: wt0

Good ifindex routes:

[2024-08-17 18:12:17] Route added: DestinationPrefix: 255.255.255.255/32, NextHop: 0.0.0.0, InterfaceIndex: 37, InterfaceAlias: wt0
[2024-08-17 18:12:17] Route added: DestinationPrefix: 224.0.0.0/4, NextHop: 0.0.0.0, InterfaceIndex: 37, InterfaceAlias: wt0
[2024-08-17 18:12:17] Route added: DestinationPrefix: 100.105.255.255/32, NextHop: 0.0.0.0, InterfaceIndex: 37, InterfaceAlias: wt0
[2024-08-17 18:12:17] Route added: DestinationPrefix: 100.105.102.132/32, NextHop: 0.0.0.0, InterfaceIndex: 37, InterfaceAlias: wt0
[2024-08-17 18:12:17] Route added: DestinationPrefix: 100.105.0.0/16, NextHop: 0.0.0.0, InterfaceIndex: 37, InterfaceAlias: wt0
[2024-08-17 18:12:17] Route added: DestinationPrefix: 198.51.100.3/32, NextHop: 10.8.2.1, InterfaceIndex: 16, InterfaceAlias: WLAN
[2024-08-17 18:12:17] Route added: DestinationPrefix: ff00::/8, NextHop: ::, InterfaceIndex: 37, InterfaceAlias: wt0
[2024-08-17 18:12:22] Route added: DestinationPrefix: 198.51.100.1/32, NextHop: 10.8.2.1, InterfaceIndex: 16, InterfaceAlias: WLAN
[2024-08-17 18:12:23] Route added: DestinationPrefix: 128.0.0.0/1, NextHop: 0.0.0.0, InterfaceIndex: 37, InterfaceAlias: wt0
[2024-08-17 18:12:23] Route added: DestinationPrefix: 127.255.255.255/32, NextHop: 0.0.0.0, InterfaceIndex: 37, InterfaceAlias: wt0
[2024-08-17 18:12:23] Route added: DestinationPrefix: 8000::/1, NextHop: ::, InterfaceIndex: 37, InterfaceAlias: wt0
[2024-08-17 18:12:23] Route added: DestinationPrefix: ::/1, NextHop: ::, InterfaceIndex: 37, InterfaceAlias: wt0

Get-NetAdapter with the issue

PS C:\WINDOWS\system32> Get-NetAdapter

Name                      InterfaceDescription                    ifIndex Status       MacAddress             LinkSpeed
----                      --------------------                    ------- ------       ----------             ---------
WLAN                      Intel(R) Wi-Fi 6 AX200 160MHz                16 Up           6C-A1-00-53-DA-B2       300 Mbps
vEthernet (WLAN)          Hyper-V Virtual Ethernet Adapter #3           80 Disconnected 6C-A1-00-53-DA-B2        10 Gbps

Get-NetAdapter with Bridging Enabled

PS C:\WINDOWS\system32> Get-NetAdapter

Name                      InterfaceDescription                    ifIndex Status       MacAddress             LinkSpeed
----                      --------------------                    ------- ------       ----------             ---------
vEthernet (WLAN)          Hyper-V Virtual Ethernet Adapter #3          44 Up           6C-A1-00-53-DA-B2       300 Mbps
WLAN                      Intel(R) Wi-Fi 6 AX200 160MHz                16 Up           6C-A1-00-53-DA-B2       300 Mbps
Netzwerkbrücke            Microsoft Network Adapter Multiplexo...      38 Up           6C-A1-00-53-DA-B2       300 Mbps

MonitorRouteChanges.ps1:

$logFile = "C:\route_changes.txt"
$previousRoutes = Get-NetRoute

while ($true) {
    $currentRoutes = Get-NetRoute
    $addedRoutes = Compare-Object $previousRoutes $currentRoutes -Property DestinationPrefix,NextHop,InterfaceIndex,InterfaceAlias -PassThru | Where-Object { $_.SideIndicator -eq '=>' }
    $removedRoutes = Compare-Object $previousRoutes $currentRoutes -Property DestinationPrefix,NextHop,InterfaceIndex,InterfaceAlias -PassThru | Where-Object { $_.SideIndicator -eq '<=' }

    $timestamp = Get-Date -Format "yyyy-MM-dd HH:mm:ss"

    if ($addedRoutes) {
        $addedRoutes | ForEach-Object {
            $routeInfo = "[$timestamp] Route added: DestinationPrefix: $($_.DestinationPrefix), NextHop: $($_.NextHop), InterfaceIndex: $($_.InterfaceIndex), InterfaceAlias: $($_.InterfaceAlias)"
            Add-Content -Path $logFile -Value $routeInfo
        }
    }

    if ($removedRoutes) {
        $removedRoutes | ForEach-Object {
            $routeInfo = "[$timestamp] Route removed: DestinationPrefix: $($_.DestinationPrefix), NextHop: $($_.NextHop), InterfaceIndex: $($_.InterfaceIndex), InterfaceAlias: $($_.InterfaceAlias)"
            Add-Content -Path $logFile -Value $routeInfo
        }
    }

    $previousRoutes = $currentRoutes
    Start-Sleep -Seconds 1
}
lixmal commented 1 month ago

Thanks for digging into that. We're using GetBestRoute2 to figure out the default route. So if windows reports the wrong thing that might be a bug there indeed. Let us know if you're able to reproduce it.