search

netbirdio / netbird

Connect your devices into a secure WireGuard®-based overlay network with SSO, MFA and granular access controls.
https://netbird.io
BSD 3-Clause "New" or "Revised" License
10.73k stars 483 forks source link

After 0.21.4 Update: SSH connection impossible #975

Open ykorzikowski opened 1 year ago

ykorzikowski commented 1 year ago

Describe the problem SSH Connection stucks at some point. Refering to https://serverfault.com/questions/210408/cannot-ssh-debug1-expecting-ssh2-msg-kex-dh-gex-reply.

Seems like the MTU of net-interface created by netbird is not set properly. 

3: wt0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1280 qdisc noqueue state UNKNOWN group default qlen 1000
    link/none 
    inet 100.***.***.***/16 brd 100.69.255.255 scope global wt0
       valid_lft forever preferred_lft forever

A MTU of 1280 is set after fresh reboot. 

ykorzikowski@yksolutions-m1:~ $ ssh -vvv yannik-dev
OpenSSH_9.0p1, LibreSSL 3.3.6
debug1: Reading configuration data /Users/ykorzikowski/.ssh/config
debug1: /Users/ykorzikowski/.ssh/config line 12: Applying options for yannik-dev
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 21: include /etc/ssh/ssh_config.d/* matched no files
debug1: /etc/ssh/ssh_config line 54: Applying options for *
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/Users/ykorzikowski/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/Users/ykorzikowski/.ssh/known_hosts2'
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug1: Connecting to yannik-dev.swokiz.net port 22.
debug1: Connection established.
debug1: identity file /Users/ykorzikowski/.ssh/id_rsa type 0
debug1: identity file /Users/ykorzikowski/.ssh/id_rsa-cert type -1
debug1: identity file /Users/ykorzikowski/.ssh/id_ecdsa type -1
debug1: identity file /Users/ykorzikowski/.ssh/id_ecdsa-cert type -1
debug1: identity file /Users/ykorzikowski/.ssh/id_ecdsa_sk type -1
debug1: identity file /Users/ykorzikowski/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /Users/ykorzikowski/.ssh/id_ed25519 type -1
debug1: identity file /Users/ykorzikowski/.ssh/id_ed25519-cert type -1
debug1: identity file /Users/ykorzikowski/.ssh/id_ed25519_sk type -1
debug1: identity file /Users/ykorzikowski/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /Users/ykorzikowski/.ssh/id_xmss type -1
debug1: identity file /Users/ykorzikowski/.ssh/id_xmss-cert type -1
debug1: identity file /Users/ykorzikowski/.ssh/id_dsa type -1
debug1: identity file /Users/ykorzikowski/.ssh/id_dsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_9.0
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.4p1 Debian-5+deb11u1
debug1: compat_banner: match: OpenSSH_8.4p1 Debian-5+deb11u1 pat OpenSSH* compat 0x04000000
debug3: fd 5 is O_NONBLOCK
debug1: Authenticating to yannik-dev.swokiz.net:22 as 'root'
debug3: record_hostkey: found key type ED25519 in file /Users/ykorzikowski/.ssh/known_hosts:186
debug3: load_hostkeys_file: loaded 1 keys from yannik-dev.swokiz.net
debug1: load_hostkeys: fopen /Users/ykorzikowski/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug3: order_hostkeyalgs: have matching best-preference key type ssh-ed25519-cert-v01@openssh.com, using HostkeyAlgorithms verbatim
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: sntrup761x25519-sha512@openssh.com,curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c
debug2: host key algorithms: ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,rsa-sha2-512,rsa-sha2-256
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256
debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
^C

To Reproduce Steps to reproduce the behavior:

  1. Remote should be reachable via netbird network
  2. Try to ssh to the remote

Expected behavior SSH is working.

NetBird status -d output:

ykorzikowski@***:~ $ netbird status -d
Peers detail:
 ***:
  NetBird IP: ***.***.****.***
  Public key: ***
  Status: Disconnected
  -- detail --
  Connection type: P2P
  Direct: false
  ICE candidate (Local/Remote): host/srflx
  Last connection update: *** 00:53:28

 ***:
  NetBird IP: ***.***.****.***
  Public key: ***
  Status: Connected
  -- detail --
  Connection type: P2P
  Direct: true
  ICE candidate (Local/Remote): srflx/host
  Last connection update: *** 12:02:28

 ***:
  NetBird IP: ***.***.****.***
  Public key: ***
  Status: Connected
  -- detail --
  Connection type: P2P
  Direct: true
  ICE candidate (Local/Remote): host/prflx
  Last connection update: *** 12:02:26

 ***:
  NetBird IP: ***.***.****.***
  Public key: ***
  Status: Disconnected
  -- detail --
  Connection type: P2P
  Direct: false
  ICE candidate (Local/Remote): srflx/host
  Last connection update: *** 00:53:28

 ***:
  NetBird IP: ***.***.****.***
  Public key: ***
  Status: Connected
  -- detail --
  Connection type: P2P
  Direct: true
  ICE candidate (Local/Remote): srflx/host
  Last connection update: *** 12:02:26

 ***:
  NetBird IP: ***.***.****.***
  Public key: ***
  Status: Connected
  -- detail --
  Connection type: P2P
  Direct: true
  ICE candidate (Local/Remote): srflx/srflx
  Last connection update: *** 12:02:25

 ***:
  NetBird IP: ***.***.****.***
  Public key: ***
  Status: Disconnected
  -- detail --
  Connection type: P2P
  Direct: false
  ICE candidate (Local/Remote): host/host
  Last connection update: *** 00:53:28

 ***:
  NetBird IP: ***.***.****.***
  Public key: ***
  Status: Disconnected
  -- detail --
  Connection type: P2P
  Direct: false
  ICE candidate (Local/Remote): host/srflx
  Last connection update: *** 00:53:28

 ***:
  NetBird IP: ***.***.****.***
  Public key: ***
  Status: Disconnected
  -- detail --
  Connection type: P2P
  Direct: false
  ICE candidate (Local/Remote): srflx/host
  Last connection update: *** 00:53:28

 ***:
  NetBird IP: ***.***.****.***
  Public key: ***
  Status: Connected
  -- detail --
  Connection type: P2P
  Direct: true
  ICE candidate (Local/Remote): host/prflx
  Last connection update: *** 12:02:27

 ***:
  NetBird IP: ***.***.****.***
  Public key: ***
  Status: Connected
  -- detail --
  Connection type: P2P
  Direct: true
  ICE candidate (Local/Remote): host/host
  Last connection update: *** 12:02:26

 ***:
  NetBird IP: ***.***.****.***
  Public key: ***
  Status: Connected
  -- detail --
  Connection type: P2P
  Direct: true
  ICE candidate (Local/Remote): host/host
  Last connection update: *** 12:02:27

 ***:
  NetBird IP: ***.***.****.***
  Public key: ***
  Status: Connected
  -- detail --
  Connection type: P2P
  Direct: true
  ICE candidate (Local/Remote): host/host
  Last connection update: *** 12:02:26

 ***:
  NetBird IP: ***.***.****.***
  Public key: ***
  Status: Connected
  -- detail --
  Connection type: Relayed
  Direct: true
  ICE candidate (Local/Remote): host/relay
  Last connection update: *** 12:02:27

 ***:
  NetBird IP: ***.***.****.***
  Public key: ***
  Status: Connected
  -- detail --
  Connection type: P2P
  Direct: true
  ICE candidate (Local/Remote): host/host
  Last connection update: *** 12:02:28

 ***:
  NetBird IP: ***.***.****.***
  Public key: ***
  Status: Connected
  -- detail --
  Connection type: P2P
  Direct: true
  ICE candidate (Local/Remote): srflx/prflx
  Last connection update: *** 12:02:28

 ***:
  NetBird IP: ***.***.****.***
  Public key: ***
  Status: Connected
  -- detail --
  Connection type: P2P
  Direct: true
  ICE candidate (Local/Remote): host/host
  Last connection update: *** 12:02:28

 ***:
  NetBird IP: ***.***.****.***
  Public key: ***
  Status: Connected
  -- detail --
  Connection type: P2P
  Direct: true
  ICE candidate (Local/Remote): srflx/host
  Last connection update: *** 12:02:27

 ***:
  NetBird IP: ***.***.****.***
  Public key: ***
  Status: Connected
  -- detail --
  Connection type: P2P
  Direct: true
  ICE candidate (Local/Remote): host/host
  Last connection update: *** 12:02:28

 ***:
  NetBird IP: ***.***.****.***
  Public key: ***
  Status: Connected
  -- detail --
  Connection type: P2P
  Direct: true
  ICE candidate (Local/Remote): srflx/host
  Last connection update: *** 12:02:25

 ***:
  NetBird IP: ***.***.****.***
  Public key: ***
  Status: Connected
  -- detail --
  Connection type: P2P
  Direct: true
  ICE candidate (Local/Remote): host/srflx
  Last connection update: *** 12:02:26

 ***:
  NetBird IP: ***.***.****.***
  Public key: ***
  Status: Connected
  -- detail --
  Connection type: P2P
  Direct: true
  ICE candidate (Local/Remote): srflx/host
  Last connection update: *** 12:02:29

 ***:
  NetBird IP: ***.***.****.***
  Public key: ***
  Status: Disconnected
  -- detail --
  Connection type: P2P
  Direct: false
  ICE candidate (Local/Remote): host/host
  Last connection update: *** 00:53:28

 ***:
  NetBird IP: ***.***.****.***
  Public key: ***
  Status: Connected
  -- detail --
  Connection type: P2P
  Direct: true
  ICE candidate (Local/Remote): host/srflx
  Last connection update: *** 12:02:26

 ***:
  NetBird IP: ***.***.****.***
  Public key: ***
  Status: Disconnected
  -- detail --
  Connection type: P2P
  Direct: false
  ICE candidate (Local/Remote): host/prflx
  Last connection update: *** 00:53:28

 ***:
  NetBird IP: ***.***.****.***
  Public key: ***
  Status: Connected
  -- detail --
  Connection type: P2P
  Direct: true
  ICE candidate (Local/Remote): host/host
  Last connection update: *** 12:02:24

Daemon version: ***
CLI version: ***
Management: Connected to https://***.***.***
Signal: Connected to https://***.***.***
FQDN: ***
NetBird IP: ***.***.****.***/16
Interface type: Userspace
Peers count: 19/26 Connected

Screenshots Not applicable

Additional context Can be fixed by: 

ip li set mtu 1200 dev wt0
pascal-fischer commented 1 year ago

Hi @ykorzikowski,

Thanks for letting us know. I will look into this. I have 2 questions upfront:

  1. Do you use Netbird's ssh feature or the systems own ssh server?
  2. What's the MTU on your local device?
ykorzikowski commented 1 year ago
  1. no, its own ssh server
  2. on my macbook
utun100: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1280
    inet 100.69.**.** --> 100.69.**.** netmask 0xff000000

I downgraded the client to 0.21.0, but issue still there. So maybe its on the servers side? I upgraded this week from 0.19.0 to 0.21.4.

Check also https://github.com/netbirdio/netbird/issues/969. I tried to run it without reverse proxy. So Management API is without now, but Signal gRPC is still proxied by nginx.

oskardotglobal commented 1 year ago

Did you try updating the server? I've been running the latest server with clients ranging from v11 to v21 behind traefik and never had issues with this.