search

netbirdio / netbird

Connect your devices into a secure WireGuard®-based overlay network with SSO, MFA and granular access controls.
https://netbird.io
BSD 3-Clause "New" or "Revised" License
11.13k stars 512 forks source link

Unable to connect between peers #1411

Open Horus-K opened 10 months ago

Horus-K commented 10 months ago

NETBIRD_DASHBOARD_TAG="v1.17.13" NETBIRD_SIGNAL_TAG="0.25.2" NETBIRD_MANAGEMENT_TAG="0.25.2" COTURN_TAG="latest"

Unable to connect between peers

[root@iZbp1imzcyvws0523mzrg4Z ~]# ping 10.255.249.205
PING 10.255.249.205 (10.255.249.205) 56(84) bytes of data.
ping: sendmsg: Operation not permitted
ping: sendmsg: Operation not permitted
ping: sendmsg: Operation not permitted
ping: sendmsg: Operation not permitted
ping: sendmsg: Operation not permitted
ping: sendmsg: Operation not permitted

acl is default !!

network routes is effic !!

image

root@iZbp11fpsa4uaxkx6jwliuZ:~# docker logs -f 666e48d2aa05
0: (1): INFO: System cpu num is 2
0: (1): INFO: log file opened: /var/tmp/turn_1_2023-12-27.log
0: (1): INFO: System enable num is 1
0: (1): INFO: Coturn Version Coturn-4.6.2 'Gorst'
0: (1): INFO: Coturn Version Coturn-4.6.2 'Gorst'
0: (1): INFO: Max number of open files/sockets allowed for this process: 1048576
0: (1): INFO: Due to the open files/sockets limitation, max supported number of TURN Sessions possible is: 524000 (approximately)
0: (1): INFO: 

==== Show him the instruments, Practical Frost: ====

0: (1): INFO: OpenSSL compile-time version: OpenSSL 3.0.11 19 Sep 2023 (0x300000b0)
0: (1): INFO: TLS 1.3 supported
0: (1): INFO: DTLS 1.2 supported
0: (1): INFO: TURN/STUN ALPN supported
0: (1): INFO: Third-party authorization (oAuth) supported
0: (1): INFO: GCM (AEAD) supported
0: (1): INFO: SQLite supported, default database location is /var/lib/coturn/turndb
0: (1): INFO: Redis supported
0: (1): INFO: PostgreSQL supported
0: (1): INFO: MySQL supported
0: (1): INFO: MongoDB supported
0: (1): INFO: Default Net Engine version: 3 (UDP thread per CPU core)
0: (1): INFO: Domain name: netbird.xxxxx.cn
0: (1): INFO: Default realm: wiretrustee.com
0: (1): WARNING: cannot find certificate file: /etc/coturn/certs/cert.pem (1)
0: (1): WARNING: cannot start TLS and DTLS listeners because certificate file is not set properly
0: (1): WARNING: cannot find private key file: /etc/coturn/private/privkey.pem (1)
0: (1): WARNING: cannot start TLS and DTLS listeners because private key file is not set properly
0: (1): INFO: Certificate file found: /etc/coturn/certs/cert.pem
0: (1): INFO: Private key file found: /etc/coturn/private/privkey.pem
0: (1): WARNING: NO EXPLICIT LISTENER ADDRESS(ES) ARE CONFIGURED
0: (1): INFO: ===========Discovering listener addresses: =========
0: (1): INFO: Listener address to use: 127.0.0.1
0: (1): INFO: Listener address to use: 192.168.14.7
0: (1): INFO: Listener address to use: 172.17.0.1
0: (1): INFO: Listener address to use: 172.24.0.1
0: (1): INFO: Listener address to use: ::1
0: (1): INFO: =====================================================
0: (1): INFO: Total: 3 'real' addresses discovered
0: (1): INFO: =====================================================
0: (1): WARNING: NO EXPLICIT RELAY ADDRESS(ES) ARE CONFIGURED
0: (1): INFO: ===========Discovering relay addresses: =============
0: (1): INFO: Relay address to use: 192.168.14.7
0: (1): INFO: Relay address to use: 172.17.0.1
0: (1): INFO: Relay address to use: 172.24.0.1
0: (1): INFO: Relay address to use: ::1
0: (1): INFO: =====================================================
0: (1): INFO: Total: 4 relay addresses discovered
0: (1): INFO: =====================================================
0: (1): INFO: pid file created: /var/tmp/turnserver.pid
0: (1): INFO: IO method: epoll (with changelist)
0: (1): INFO: Wait for relay ports initialization...
0: (1): INFO:   relay 192.168.14.7 initialization...
0: (1): INFO:   relay 192.168.14.7 initialization done
0: (1): INFO:   relay 172.17.0.1 initialization...
0: (1): INFO:   relay 172.17.0.1 initialization done
0: (1): INFO:   relay 172.24.0.1 initialization...
0: (1): INFO:   relay 172.24.0.1 initialization done
0: (1): INFO:   relay ::1 initialization...
0: (1): INFO:   relay ::1 initialization done
0: (1): INFO: Relay ports initialization done
0: (1): INFO: Total General servers: 2
9: (9): DEBUG: turn server id=1 created
9: (8): DEBUG: turn server id=0 created
9: (1): INFO: Total auth threads: 3
9: (1): INFO: prometheus collector disabled, not started
9: (8): ERROR: check_stun_auth: user self credentials are incorrect
9: (9): ERROR: check_stun_auth: user self credentials are incorrect
34: (9): ERROR: check_stun_auth: user self credentials are incorrect
35: (8): ERROR: check_stun_auth: user self credentials are incorrect
51: (9): ERROR: check_stun_auth: user self credentials are incorrect
52: (9): ERROR: check_stun_auth: user self credentials are incorrect
52: (9): ERROR: check_stun_auth: user self credentials are incorrect
52: (8): ERROR: check_stun_auth: user self credentials are incorrect
52: (9): ERROR: check_stun_auth: user self credentials are incorrect
53: (9): ERROR: check_stun_auth: user self credentials are incorrect
53: (9): ERROR: check_stun_auth: user self credentials are incorrect
53: (8): ERROR: check_stun_auth: user self credentials are incorrect
53: (9): ERROR: check_stun_auth: user self credentials are incorrect
57: (8): ERROR: check_stun_auth: user self credentials are incorrect
62: (8): ERROR: check_stun_auth: user self credentials are incorrect
63: (8): ERROR: check_stun_auth: user self credentials are incorrect
65: (8): ERROR: check_stun_auth: user self credentials are incorrect
65: (8): ERROR: check_stun_auth: user self credentials are incorrect
79: (9): ERROR: check_stun_auth: user self credentials are incorrect
79: (8): ERROR: check_stun_auth: user self credentials are incorrect
79: (9): ERROR: check_stun_auth: user self credentials are incorrect
80: (8): ERROR: check_stun_auth: user self credentials are incorrect
80: (8): ERROR: check_stun_auth: user self credentials are incorrect
80: (8): ERROR: check_stun_auth: user self credentials are incorrect
80: (9): ERROR: check_stun_auth: user self credentials are incorrect
80: (9): ERROR: check_stun_auth: user self credentials are incorrect
80: (8): ERROR: check_stun_auth: user self credentials are incorrect
Horus-K commented 10 months ago 

[root@iZbp1imzcyvws0523mzrg4Z ~]# netbird status
Daemon version: 0.24.4
CLI version: 0.24.4
Management: Connected
Signal: Connected
FQDN: ss-pre1.netbird.selfhosted
NetBird IP: 10.255.248.87/22
Interface type: Userspace
Peers count: 10/22 Connected

C:\Users\vvv>netbird status
Daemon version: 0.25.2
CLI version: 0.25.2
Management: Connected
Signal: Connected
FQDN: qh.netbird.selfhosted
NetBird IP: 10.255.249.205/22
Interface type: Userspace
Peers count: 10/22 Connected
bcmmbaga commented 10 months ago

Hello @Horus-K, Thank you for reporting the issue. To assist us in diagnosing and resolving the problem, could you please share the following information:

The detailed Netbird status from both nodes using the command: netbird status --detail

Additionally, provide the firewall rules from the node where the attempted ping occurred. You can obtain this information with the command: sudo nft list ruleset.

Horus-K commented 10 months ago 
[root@iZbp1imzcyvws0523mzrg4Z ~]# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
CILIUM_INPUT  all  --  anywhere             anywhere             /* cilium-feeder: CILIUM_INPUT */
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:openvpn
ACCEPT     udp  --  anywhere             iZbp1imzcyvws0523mzrg4Z  udp dpt:domain
ACCEPT     tcp  --  anywhere             iZbp1imzcyvws0523mzrg4Z  tcp dpt:domain
KUBE-FIREWALL  all  --  anywhere             anywhere            
ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:openvpn
ACCEPT     all  --  anywhere             anywhere             state RELATED,ESTABLISHED
ACCEPT     all  --  10.255.248.0/22      anywhere            
ACCEPT     icmp --  anywhere             anywhere             icmp echo-request state NEW,RELATED,ESTABLISHED
ACCEPT     all  --  10.255.248.0/22      anywhere            

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
CILIUM_FORWARD  all  --  anywhere             anywhere             /* cilium-feeder: CILIUM_FORWARD */
KUBE-FORWARD  all  --  anywhere             anywhere             /* kubernetes forwarding rules */
DOCKER-USER  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
CILIUM_OUTPUT  all  --  anywhere             anywhere             /* cilium-feeder: CILIUM_OUTPUT */
ACCEPT     udp  --  iZbp1imzcyvws0523mzrg4Z  anywhere             udp spt:domain
ACCEPT     tcp  --  iZbp1imzcyvws0523mzrg4Z  anywhere             tcp spt:domain
KUBE-FIREWALL  all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             10.255.248.0/22     
ACCEPT     icmp --  anywhere             anywhere             icmp echo-reply state RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             10.255.248.0/22     

Chain CILIUM_FORWARD (1 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere             /* cilium: any->cluster on cilium_host forward accept */
ACCEPT     all  --  anywhere             anywhere             /* cilium: cluster->any on cilium_host forward accept (nodeport) */
ACCEPT     all  --  anywhere             anywhere             /* cilium: cluster->any on lxc+ forward accept */
ACCEPT     all  --  anywhere             anywhere             /* cilium: cluster->any on cilium_net forward accept (nodeport) */

Chain CILIUM_INPUT (1 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere             mark match 0x200/0xf00 /* cilium: ACCEPT for proxy traffic */

Chain CILIUM_OUTPUT (1 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere             mark match 0xa00/0xfffffeff /* cilium: ACCEPT for proxy return traffic */
MARK       all  --  anywhere             anywhere             mark match ! 0xe00/0xf00 mark match ! 0xd00/0xf00 mark match ! 0xa00/0xe00 /* cilium: host->any mark as from host */ MARK xset 0xc00/0xf00

Chain DOCKER (1 references)
target     prot opt source               destination         

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination         
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-USER (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere            

Chain KUBE-FIREWALL (2 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere             /* kubernetes firewall for dropping marked packets */ mark match 0x8000/0x8000
DROP       all  -- !loopback/8           loopback/8           /* block incoming localnet connections */ ! ctstate RELATED,ESTABLISHED,DNAT

Chain KUBE-FORWARD (1 references)
target     prot opt source               destination         
ACCEPT     all  --  anywhere             anywhere             /* kubernetes forwarding rules */ mark match 0x4000/0x4000
ACCEPT     all  --  anywhere             anywhere             /* kubernetes forwarding conntrack pod source rule */ ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere             /* kubernetes forwarding conntrack pod destination rule */ ctstate RELATED,ESTABLISHED

Chain KUBE-KUBELET-CANARY (0 references)
target     prot opt source               destination
Horus-K commented 10 months ago

[root@iZbp1imzcyvws0523mzrg4Z ~]# netbird status --detail Peers detail:

qh.netbird.selfhosted: NetBird IP: 10.255.249.205 Public key: Status: Connected -- detail -- Connection type: P2P Direct: true ICE candidate (Local/Remote): srflx/srflx Last connection update: 2023-12-28 16:21:50

Daemon version: 0.25.2 CLI version: 0.25.2 Management: Connected to https://netbird.xxxx.cn:33073 Signal: Connected to http://netbird.xxxx.cn:10000 FQDN: ss-pre1.netbird.selfhosted NetBird IP: 10.255.248.87/22 Interface type: Userspace Peers count: 12/22 Connected

C:\Windows\system32>netbird status -d Peers detail: ss-pre1.netbird.selfhosted: NetBird IP: 10.255.248.87 Public key: xx Status: Connected -- detail -- Connection type: P2P Direct: true ICE candidate (Local/Remote): srflx/srflx Last connection update: 2023-12-28 16:22:01

Daemon version: 0.25.2 CLI version: 0.25.2 Management: Connected to https://netbird.xxxx.cn:33073 Signal: Connected to http://netbird.xxxx.cn:10000 FQDN: qh.netbird.selfhosted NetBird IP: 10.255.249.205/22 Interface type: Userspace Peers count: 12/22 Connected

mlsmaycon commented 10 months ago

@Horus-K, we asked a few more questions via Slack; it might be better to troubleshoot there.

PavelNiedoba commented 9 months ago

Nobody else will benefit from slack advice, google does not index it and slack free account does not show anything older than 3months

vulndev commented 7 months ago

I have the exact same Problem: Mobile Devices can not connect to devices which are not directly accessable over the internet e.g. behind a firewall. The TURN server seems not to step in and it might be because of this error:

ERROR: check_stun_auth: user self credentials are incorrect

The username and password in management.json and turnserver.conf are the same. I checked them also within the containers. If there have been a solution for this problem here it might help me also.

vulndev commented 7 months ago

I troubleshooted the problem and found out, that my problem was the coturn server which is also placed behind a NAT gateway with dynamic ip addresses. Therefor the external_ipparameter in /etc/turnserver.conf can not be filled but should be. I wrote a little script which updates the external_ip statement whenever the public address changed:

#!/bin/bash

# path to turnserver.conf
TURN_CONF="/etc/turnserver.conf"

# get external ip address
EXTERNAL_IP=$(curl -s ifconfig.me)

# read current ip address from turnserver.conf
CURRENT_IP=$(grep "^external-ip" "$TURN_CONF" | awk -F"=" '{print $2}')

# check if public and current ip address differ
if [ "$EXTERNAL_IP" != "$CURRENT_IP" ]; then
    # set new public ip address
    sed -i "s/^external-ip.*/external-ip=$EXTERNAL_IP/" "$TURN_CONF"

    # restart coturn
    systemctl restart coturn
    echo "Coturn was restarted with a new external ip address: ($EXTERNAL_IP)" | logger
else
    echo "The external ip address has not changed." | logger
fi

Whomever this may help.