netbirdio / netbird

Connect your devices into a single secure private WireGuard®-based mesh network with SSO/MFA and simple access controls.
https://netbird.io
BSD 3-Clause "New" or "Revised" License
9.81k stars 426 forks source link

Can only ping hosts on same network on self-hosted on Oracle #2120

Open EDIflyer opened 3 weeks ago

EDIflyer commented 3 weeks ago

Describe the problem Can only ping other Netbird hosts on the same local network

To Reproduce

A clear and concise description of what you expected to happen. Pinging to work between networks

Are you using NetBird Cloud? No, using selfhosted

NetBird version 0.27.10

Additional context All hosts are showing as green on the Netbird dashboard. When running status -dA I note the clients I can't ping are showing as offline in that output despite being online on the dashboard.

NetBird status -dA output


Peers detail:
 pixel8.netbird.selfhosted:
  NetBird IP: 100.72.18.3
  Public key: spdafpxNQ9EeM3tPZq3J59T50P5C3/qnxFD72ZaLbQg=
  Status: Disconnected
  -- detail --
  Connection type: P2P
  Direct: false
  ICE candidate (Local/Remote): host/host
  ICE candidate endpoints (Local/Remote): 192.168.1.24:51820/192.168.1.96:51820
  Last connection update: 19 hours, 11 minutes ago
  Last WireGuard handshake: 2 minutes, 4 seconds ago
  Transfer status (received/sent) 5.9 KiB/1.8 KiB
  Quantum resistance: false
  Routes: -
  Latency: 0s

 chillblast-edi.netbird.selfhosted:
  NetBird IP: 100.72.63.91
  Public key: VZfY5WgbKibJNCU/WzMqoX8hBn9iyC3YxoTqiizSmwM=
  Status: Disconnected
  -- detail --
  Connection type: P2P
  Direct: false
  ICE candidate (Local/Remote): host/host
  ICE candidate endpoints (Local/Remote): 192.168.1.24:51820/192.168.1.96:51820
  Last connection update: 19 hours, 28 minutes ago
  Last WireGuard handshake: 2 minutes, 4 seconds ago
  Transfer status (received/sent) 5.9 KiB/1.8 KiB
  Quantum resistance: false
  Routes: -
  Latency: 39.862337ms

 pi4.netbird.selfhosted:
  NetBird IP: 100.72.142.196
  Public key: xoSpZtj5S4NADpy/Ln1L9X7T2KwG3QvEX6h5/04onwQ=
  Status: Connecting
  -- detail --
  Connection type: P2P
  Direct: false
  ICE candidate (Local/Remote): host/host
  ICE candidate endpoints (Local/Remote): 192.168.1.24:51820/192.168.1.96:51820
  Last connection update: 8 seconds ago
  Last WireGuard handshake: 2 minutes, 4 seconds ago
  Transfer status (received/sent) 5.9 KiB/1.8 KiB
  Quantum resistance: false
  Routes: -
  Latency: 0s

 chillblast-dnd.netbird.selfhosted:
  NetBird IP: 100.72.158.226
  Public key: BgIDgECerz62sgaC4U4dSLW8MvEXGBr0PGu62TH7bwA=
  Status: Connected
  -- detail --
  Connection type: P2P
  Direct: true
  ICE candidate (Local/Remote): host/host
  ICE candidate endpoints (Local/Remote): 192.168.1.24:51820/192.168.1.96:51820
  Last connection update: 37 minutes, 30 seconds ago
  Last WireGuard handshake: 2 minutes, 4 seconds ago
  Transfer status (received/sent) 5.9 KiB/1.8 KiB
  Quantum resistance: false
  Routes: -
  Latency: 1.614116ms

OS: linux/arm (ARMv)
Daemon version: 0.27.10
CLI version: 0.27.10
Management: Connected to <https://network.anon-aHrXN.domain:443>
Signal: Connected to <https://network.anon-aHrXN.domain:443>
Relays:
  [stun:network.anon-aHrXN.domain:3478] is Unavailable, reason: stun request: context deadline exceeded
  [turn:network.anon-aHrXN.domain:3478?transport=udp] is Unavailable, reason: allocate: all retransmissions failed for xIkecmxLzfjXr1y/
Nameservers:
FQDN: pi400.netbird.selfhosted
NetBird IP: 100.72.103.36/16
Interface type: Kernel
Quantum resistance: false
Routes: -
Peers count: 1/4 Connected```

[Slack Message](https://netbirdio.slack.com/archives/C05T5K65X7U/p1718045885486609)
mlsmaycon commented 2 weeks ago

@EDIflyer it seems like your relay service is not reachable. Can you run the test from this page? https://docs.netbird.io/selfhosted/troubleshooting

EDIflyer commented 2 weeks ago

Hi @mlsmaycon sorry for the slow reply, been tied up with work. Have tried it out and I get the following (server name redacted)...

Note: errors from onicecandidateerror above are not necessarily fatal. For example an IPv6 DNS lookup may fail but relay candidates can still be gathered via IPv4.
The server stun:netbird.<MYDOMAIN.COM>:3478 returned an error with code=701:
STUN host lookup received error.
The server turn:netbird.<MYDOMAIN.COM>:3478?transport=udp returned an error with code=701:
TURN host lookup received error.
The server stun:netbird.<MYDOMAIN.COM>:3478 returned an error with code=701:
STUN binding request timed out.
The server turn:netbird.<MYDOMAIN.COM>:3478?transport=udp returned an error with code=701:
TURN allocate request timed out.

As far as I can tell from sudo ss -atpu I have connections coming in port 3478 OK?

Netid         State             Recv-Q         Send-Q                    Local Address:Port                        Peer Address:Port          Process
udp           UNCONN            0              0                         127.0.0.53%lo:domain                           0.0.0.0:*              users:(("systemd-resolve",pid=657,fd=13))
udp           UNCONN            0              0                      10.0.0.67%enp0s6:bootpc                           0.0.0.0:*              users:(("systemd-network",pid=655,fd=15))
udp           UNCONN            0              0                               0.0.0.0:sunrpc                           0.0.0.0:*              users:(("rpcbind",pid=594,fd=5),("systemd",pid=1,fd=141))
udp           UNCONN            0              0                            172.18.0.1:3478                             0.0.0.0:*              users:(("turnserver",pid=4175,fd=32))
udp           UNCONN            0              0                            172.18.0.1:3478                             0.0.0.0:*              users:(("turnserver",pid=4175,fd=31))
udp           UNCONN            0              0                            172.17.0.1:3478                             0.0.0.0:*              users:(("turnserver",pid=4175,fd=30))
udp           UNCONN            0              0                            172.17.0.1:3478                             0.0.0.0:*              users:(("turnserver",pid=4175,fd=29))
udp           UNCONN            0              0                             10.0.0.67:3478                             0.0.0.0:*              users:(("turnserver",pid=4175,fd=28))
udp           UNCONN            0              0                             10.0.0.67:3478                             0.0.0.0:*              users:(("turnserver",pid=4175,fd=27))
udp           UNCONN            0              0                             127.0.0.1:3478                             0.0.0.0:*              users:(("turnserver",pid=4175,fd=26))
udp           UNCONN            0              0                             127.0.0.1:3478                             0.0.0.0:*              users:(("turnserver",pid=4175,fd=25))
udp           UNCONN            0              0                                  [::]:sunrpc                              [::]:*              users:(("rpcbind",pid=594,fd=7),("systemd",pid=1,fd=143))
udp           UNCONN            0              0                                 [::1]:3478                                [::]:*              users:(("turnserver",pid=4175,fd=33))
udp           UNCONN            0              0                                 [::1]:3478                                [::]:*              users:(("turnserver",pid=4175,fd=34))
tcp           LISTEN            0              1024                         172.17.0.1:3478                             0.0.0.0:*              users:(("turnserver",pid=4175,fd=22))
tcp           LISTEN            0              1024                         172.17.0.1:3478                             0.0.0.0:*              users:(("turnserver",pid=4175,fd=13))
tcp           LISTEN            0              1024                          127.0.0.1:3478                             0.0.0.0:*              users:(("turnserver",pid=4175,fd=20))
tcp           LISTEN            0              1024                          127.0.0.1:3478                             0.0.0.0:*              users:(("turnserver",pid=4175,fd=11))
tcp           LISTEN            0              128                             0.0.0.0:ssh                              0.0.0.0:*              users:(("sshd",pid=780,fd=3))
tcp           LISTEN            0              4096                            0.0.0.0:http                             0.0.0.0:*              users:(("docker-proxy",pid=3423,fd=4))
tcp           LISTEN            0              4096                            0.0.0.0:sunrpc                           0.0.0.0:*              users:(("rpcbind",pid=594,fd=4),("systemd",pid=1,fd=140))
tcp           LISTEN            0              4096                            0.0.0.0:https                            0.0.0.0:*              users:(("docker-proxy",pid=3404,fd=4))
tcp           LISTEN            0              1024                          10.0.0.67:3478                             0.0.0.0:*              users:(("turnserver",pid=4175,fd=21))
tcp           LISTEN            0              1024                          10.0.0.67:3478                             0.0.0.0:*              users:(("turnserver",pid=4175,fd=12))
tcp           LISTEN            0              1024                         172.18.0.1:3478                             0.0.0.0:*              users:(("turnserver",pid=4175,fd=23))
tcp           LISTEN            0              1024                         172.18.0.1:3478                             0.0.0.0:*              users:(("turnserver",pid=4175,fd=14))
tcp           LISTEN            0              4096                            0.0.0.0:http-alt                         0.0.0.0:*              users:(("docker-proxy",pid=3385,fd=4))
tcp           LISTEN            0              4096                      127.0.0.53%lo:domain                           0.0.0.0:*              users:(("systemd-resolve",pid=657,fd=14))
tcp           ESTAB             0              0                             10.0.0.67:ssh                        (redacted):54251          users:(("sshd",pid=826757,fd=4),("sshd",pid=826669,fd=4))
tcp           LISTEN            0              128                                [::]:ssh                                 [::]:*              users:(("sshd",pid=780,fd=4))
tcp           LISTEN            0              4096                               [::]:http                                [::]:*              users:(("docker-proxy",pid=3430,fd=4))
tcp           LISTEN            0              4096                               [::]:sunrpc                              [::]:*              users:(("rpcbind",pid=594,fd=6),("systemd",pid=1,fd=142))
tcp           LISTEN            0              4096                               [::]:https                               [::]:*              users:(("docker-proxy",pid=3410,fd=4))
tcp           LISTEN            0              1024                              [::1]:3478                                [::]:*              users:(("turnserver",pid=4175,fd=15))
tcp           LISTEN            0              1024                              [::1]:3478                                [::]:*              users:(("turnserver",pid=4175,fd=24))
tcp           LISTEN            0              4096                               [::]:http-alt                            [::]:*              users:(("docker-proxy",pid=3392,fd=4))

and from sudo iptables --list...

Chain INPUT (policy ACCEPT)
target     prot opt source               destination
ACCEPT     udp  --  anywhere             anywhere             udp dpt:3478
ACCEPT     all  --  anywhere             anywhere             state RELATED,ESTABLISHED
ACCEPT     icmp --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     udp  --  anywhere             anywhere             udp spt:ntp
ACCEPT     tcp  --  anywhere             anywhere             state NEW tcp dpt:ssh
REJECT     all  --  anywhere             anywhere             reject-with icmp-host-prohibited

Chain FORWARD (policy DROP)
target     prot opt source               destination
DOCKER-USER  all  --  anywhere             anywhere
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
REJECT     all  --  anywhere             anywhere             reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
InstanceServices  all  --  anywhere             link-local/16

Chain DOCKER (2 references)
target     prot opt source               destination
ACCEPT     tcp  --  anywhere             172.18.0.3           tcp dpt:http-alt
ACCEPT     tcp  --  anywhere             172.18.0.3           tcp dpt:https
ACCEPT     tcp  --  anywhere             172.18.0.3           tcp dpt:http

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere
RETURN     all  --  anywhere             anywhere

Chain DOCKER-ISOLATION-STAGE-2 (2 references)
target     prot opt source               destination
DROP       all  --  anywhere             anywhere
DROP       all  --  anywhere             anywhere
RETURN     all  --  anywhere             anywhere

Chain DOCKER-USER (1 references)
target     prot opt source               destination
RETURN     all  --  anywhere             anywhere

Chain InstanceServices (1 references)
target     prot opt source               destination
ACCEPT     tcp  --  anywhere             169.254.0.2          owner UID match root tcp dpt:iscsi-target /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
ACCEPT     tcp  --  anywhere             169.254.2.0/24       owner UID match root tcp dpt:iscsi-target /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
ACCEPT     tcp  --  anywhere             169.254.4.0/24       owner UID match root tcp dpt:iscsi-target /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
ACCEPT     tcp  --  anywhere             169.254.5.0/24       owner UID match root tcp dpt:iscsi-target /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
ACCEPT     tcp  --  anywhere             169.254.0.2          tcp dpt:http /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
ACCEPT     udp  --  anywhere             169.254.169.254      udp dpt:domain /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
ACCEPT     tcp  --  anywhere             169.254.169.254      tcp dpt:domain /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
ACCEPT     tcp  --  anywhere             169.254.0.3          owner UID match root tcp dpt:http /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
ACCEPT     tcp  --  anywhere             169.254.0.4          tcp dpt:http /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
ACCEPT     tcp  --  anywhere             169.254.169.254      tcp dpt:http /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
ACCEPT     udp  --  anywhere             169.254.169.254      udp dpt:bootps /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
ACCEPT     udp  --  anywhere             169.254.169.254      udp dpt:tftp /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
ACCEPT     udp  --  anywhere             169.254.169.254      udp dpt:ntp /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */
REJECT     tcp  --  anywhere             link-local/16        tcp /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */ reject-with tcp-reset
REJECT     udp  --  anywhere             link-local/16        udp /* See the Oracle-Provided Images section in the Oracle Cloud Infrastructure documentation for security impact of modifying or removing this rule */ reject-with icmp-port-unreachable