search

labgrid-project / labgrid

Embedded systems control library for development, testing and installation
https://labgrid.readthedocs.io/
Other
339 stars 176 forks source link

SSHDriver: Ubuntu 20.04 Clients getting stuck when trying to ssh into target via exporter proxy #1325

Closed ep1cman closed 9 months ago

ep1cman commented 9 months ago

We currently have the following setup:

On Ubuntu 22.04 (OpenSSH 8.9) and MacOS (OpenSSH 9.3) clients. This works as expected.

On Ubuntu 20.04 (OpenSSH 8.2) clients however stall when attempting to run:

labgrid-client -p PLACE_NAME ssh

Investigations reveal it is getting stuck here: https://github.com/labgrid-project/labgrid/blob/2d6f93c0e298442453603f2fd5430da37606da09/labgrid/driver/sshdriver.py#L151

Commenting out the following line seems to resolve the issue

https://github.com/labgrid-project/labgrid/blob/2d6f93c0e298442453603f2fd5430da37606da09/labgrid/driver/sshdriver.py#L142

Logs from a working machine:

debug1: Reading configuration data /Users/sebgos01/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 21: include /etc/ssh/ssh_config.d/* matched no files
debug1: /etc/ssh/ssh_config line 54: Applying options for *
debug1: /etc/ssh/ssh_config line 58: Applying options for *
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug1: Executing proxy command: exec ssh -x -o LogLevel=DEBUG -o PasswordAuthentication=no -o ControlMaster=no -o ControlPath=/tmp/lg-con-mw7wuy8r/control-REDACTED REDACTED -W REDACTED 2>/tmp/lg-ssh-hyu26w_p/proxy-stderr
debug1: identity file /Users/REDACTED/.ssh/id_rsa type 0
debug1: identity file /Users/REDACTED/.ssh/id_rsa-cert type -1
debug1: identity file /Users/REDACTED/.ssh/id_ecdsa type -1
debug1: identity file /Users/REDACTED/.ssh/id_ecdsa-cert type -1
debug1: identity file /Users/REDACTED/.ssh/id_ecdsa_sk type -1
debug1: identity file /Users/REDACTED/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /Users/REDACTED/.ssh/id_ed25519 type -1
debug1: identity file /Users/REDACTED/.ssh/id_ed25519-cert type -1
debug1: identity file /Users/REDACTED/.ssh/id_ed25519_sk type -1
debug1: identity file /Users/REDACTED/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /Users/REDACTED/.ssh/id_xmss type -1
debug1: identity file /Users/REDACTED/.ssh/id_xmss-cert type -1
debug1: identity file /Users/REDACTED/.ssh/id_dsa type -1
debug1: identity file /Users/REDACTED/.ssh/id_dsa-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_9.3
debug1: Remote protocol version 2.0, remote software version OpenSSH_9.2p1 Debian-2+deb12u1
debug1: compat_banner: match: OpenSSH_9.2p1 Debian-2+deb12u1 pat OpenSSH* compat 0x04000000
debug1: Authenticating to REDACTED:22 as 'REDACTED'
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: sntrup761x25519-sha512@openssh.com
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-ed25519 SHA256:2U7BLZRRozwCEpGxLiYfmIffV29n4/n8jhBsWTGq2Aw
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
Warning: Permanently added 'REDACTED' (ED25519) to the list of known hosts.
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /Users/REDACTED/.ssh/id_rsa RSA SHA256:VSpwC/JTo3lmxSCa+AOQNXt+ptP1CDZ/kr7LvDaRofA
debug1: Will attempt key: /Users/REDACTED/.ssh/id_ecdsa 
debug1: Will attempt key: /Users/REDACTED/.ssh/id_ecdsa_sk 
debug1: Will attempt key: /Users/REDACTED/.ssh/id_ed25519 
debug1: Will attempt key: /Users/REDACTED/.ssh/id_ed25519_sk 
debug1: Will attempt key: /Users/REDACTED/.ssh/id_xmss 
debug1: Will attempt key: /Users/REDACTED/.ssh/id_dsa 
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@openssh.com,webauthn-sk-ecdsa-sha2-nistp256@openssh.com,ssh-dss,ssh-rsa,rsa-sha2-256,rsa-sha2-512>
debug1: kex_input_ext_info: publickey-hostbound@openssh.com=<0>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering public key: /Users/REDACTED/.ssh/id_rsa RSA SHA256:VSpwC/JTo3lmxSCa+AOQNXt+ptP1CDZ/kr7LvDaRofA
debug1: Authentications that can continue: publickey,password
debug1: Trying private key: /Users/REDACTED/.ssh/id_ecdsa
debug1: Trying private key: /Users/REDACTED/.ssh/id_ecdsa_sk
debug1: Trying private key: /Users/REDACTED/.ssh/id_ed25519
debug1: Trying private key: /Users/REDACTED/.ssh/id_ed25519_sk
debug1: Trying private key: /Users/REDACTED/.ssh/id_xmss
debug1: Trying private key: /Users/REDACTED/.ssh/id_dsa
debug1: Next authentication method: password
debug1: read_passphrase: requested to askpass
Authenticated to REDACTED (via proxy) using "password".
debug1: setting up multiplex master socket
debug1: channel 0: new mux listener [/tmp/lg-ssh-hyu26w_p/control-REDACTED] (inactive timeout: 0)
debug1: forking to background

Logs from a broken machine

debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 3: Applying options for *
debug1: Executing proxy command: exec ssh -x -o LogLevel=DEBUG -o PasswordAuthentication=no -o ControlMaster=no -o ControlPath=/tmp/lg-con-9gqnvwoi/control-REDACTED REDACTED -W REDACTED:22 2>/tmp/lg-ssh-8pqw0g1_/proxy-stderr
debug1: identity file /home/REDACTED/.ssh/id_rsa type 0
debug1: identity file /home/REDACTED/.ssh/id_rsa-cert type -1
debug1: identity file /home/REDACTED/.ssh/id_dsa type -1
debug1: identity file /home/REDACTED/.ssh/id_dsa-cert type -1
debug1: identity file /home/REDACTED/.ssh/id_ecdsa type -1
debug1: identity file /home/REDACTED/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/REDACTED/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/REDACTED/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/REDACTED/.ssh/id_ed25519 type -1
debug1: identity file /home/REDACTED/.ssh/id_ed25519-cert type -1
debug1: identity file /home/REDACTED/.ssh/id_ed25519_sk type -1
debug1: identity file /home/REDACTED/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/REDACTED/.ssh/id_xmss type -1
debug1: identity file /home/REDACTED/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.11
debug1: Remote protocol version 2.0, remote software version OpenSSH_9.2p1 Debian-2+deb12u1
debug1: match: OpenSSH_9.2p1 Debian-2+deb12u1 pat OpenSSH* compat 0x04000000
debug1: Authenticating to REDACTED:22 as 'REDACTED'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:XIymR1icQOTE4fpTpQrUD/wBaOfit/bSv8FWsWZpFV4
Warning: Permanently added 'REDACTED' (ECDSA) to the list of known hosts.
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /home/REDACTED/.ssh/id_rsa RSA SHA256:i0jeRpmcWxPQCvK0cMD4XZ7xMxAMlEITZfnT78DB/g0
debug1: Will attempt key: /home/REDACTED/.ssh/id_dsa 
debug1: Will attempt key: /home/REDACTED/.ssh/id_ecdsa 
debug1: Will attempt key: /home/REDACTED/.ssh/id_ecdsa_sk 
debug1: Will attempt key: /home/REDACTED/.ssh/id_ed25519 
debug1: Will attempt key: /home/REDACTED/.ssh/id_ed25519_sk 
debug1: Will attempt key: /home/REDACTED/.ssh/id_xmss 
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,sk-ssh-ed25519@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256@openssh.com,webauthn-sk-ecdsa-sha2-nistp256@openssh.com,ssh-dss,ssh-rsa,rsa-sha2-256,rsa-sha2-512>
debug1: kex_input_ext_info: publickey-hostbound@openssh.com (unrecognised)
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering public key: /home/REDACTED/.ssh/id_rsa RSA SHA256:i0jeRpmcWxPQCvK0cMD4XZ7xMxAMlEITZfnT78DB/g0
debug1: Authentications that can continue: publickey,password
debug1: Trying private key: /home/REDACTED/.ssh/id_dsa
debug1: Trying private key: /home/REDACTED/.ssh/id_ecdsa
debug1: Trying private key: /home/REDACTED/.ssh/id_ecdsa_sk
debug1: Trying private key: /home/REDACTED/.ssh/id_ed25519
debug1: Trying private key: /home/REDACTED/.ssh/id_ed25519_sk
debug1: Trying private key: /home/REDACTED/.ssh/id_xmss
debug1: Next authentication method: password
debug1: read_passphrase: can't open /dev/tty: No such device or address
debug1: Authentication succeeded (password).
Authenticated to REDACTED (via proxy).
debug1: setting up multiplex master socket
debug1: channel 0: new [/tmp/lg-ssh-8pqw0g1_/control-REDACTED]
debug1: forking to background
debug1: Entering interactive session.
debug1: pledge: id
ep1cman commented 9 months ago

https://bugzilla.mindrot.org/show_bug.cgi?id=3137

Seems to be related to this. Removing -f from the following line https://github.com/labgrid-project/labgrid/blob/2d6f93c0e298442453603f2fd5430da37606da09/labgrid/driver/sshdriver.py#L108 also resolves the issue

Bastian-Krause commented 9 months ago

Might be related to #1288.

ep1cman commented 9 months ago

Ah yes, you are correct. Looks like we had deviated from the upstream code and this didn't get pulled in. Seems to work ok now, if I face further issues I will re-open