NetworkService: Password Authentication not preferred when connecting via Proxy

[NiklasReisser]

Hi,

I have a target which is only reachable by using Labgrids Proxy-Mechanism, or (obviously) by logging into the host which connects to the target and which runs the exporter.

The target only supports password authentication.

I faced the following issue: When setting MaxAuthTries 1 in the targets sshd_config (or when having >= than MaxAuthTries' ssh identitys), I am no longer able to connect to the target via the Proxy. The direct connection (labgrid-client on the exporter host) still works fine.

To be honest, I don't know what causes this. A wild guess is its related to the environment (SSH_ASKPASS_REQUIRE) not being available on the Jump host.

A workaround is adding something like this in SSHDriver._start_own_master_once:

        if self.networkservice.password:
            args += ["-o", 'PreferredAuthentications="password"']

Here is a log with ssh's loglevel set to debug and an additional log of the full ssh command labgrid builds:

$ labgrid-client -p slot2 ssh --name eth
WARNING: RUN: ['ssh', '-f', '-o', 'LogLevel=DEBUG', '-x', '-o', 'ConnectTimeout=30', '-o', 'ControlPersist=300', '-o', 'UserKnownHostsFile=/dev/null', '-o', 'StrictHostKeyChecking=no', '-o', 'ServerAliveInterval=15', '-MN', '-S', '/tmp/labgrid-ssh-tmp-pq_s64tp/control-169.254.21.34', '-p', '22', '-l', 'root', '169.254.21.34', '-o', 'ProxyCommand=ssh -x -o LogLevel=DEBUG -o PasswordAuthentication=no -o ControlMaster=no -o ControlPath=/tmp/labgrid-connection-vce7pg2x/control-exporter exporter -W 169.254.21.34:22 2>/tmp/labgrid-ssh-tmp-pq_s64tp/proxy-stderr']
WARNING: ssh: debug1: Reading configuration data /home/devel/.ssh/config
WARNING: ssh: debug1: Reading configuration data /etc/ssh/ssh_config
WARNING: ssh: debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
WARNING: ssh: debug1: /etc/ssh/ssh_config line 21: Applying options for *
WARNING: ssh: debug1: Executing proxy command: exec ssh -x -o LogLevel=DEBUG -o PasswordAuthentication=no -o ControlMaster=no -o ControlPath=/tmp/labgrid-connection-vce7pg2x/control-exporter exporter -W 169.254.21.34:22 2>/tmp/labgrid-ssh-tmp-pq_s64tp/proxy-stderr
WARNING: ssh: debug1: identity file /home/devel/.ssh/id_rsa type 0
WARNING: ssh: debug1: identity file /home/devel/.ssh/id_rsa-cert type -1
WARNING: ssh: debug1: identity file /home/devel/.ssh/id_dsa type -1
WARNING: ssh: debug1: identity file /home/devel/.ssh/id_dsa-cert type -1
WARNING: ssh: debug1: identity file /home/devel/.ssh/id_ecdsa type -1
WARNING: ssh: debug1: identity file /home/devel/.ssh/id_ecdsa-cert type -1
WARNING: ssh: debug1: identity file /home/devel/.ssh/id_ecdsa_sk type -1
WARNING: ssh: debug1: identity file /home/devel/.ssh/id_ecdsa_sk-cert type -1
WARNING: ssh: debug1: identity file /home/devel/.ssh/id_ed25519 type -1
WARNING: ssh: debug1: identity file /home/devel/.ssh/id_ed25519-cert type -1
WARNING: ssh: debug1: identity file /home/devel/.ssh/id_ed25519_sk type -1
WARNING: ssh: debug1: identity file /home/devel/.ssh/id_ed25519_sk-cert type -1
WARNING: ssh: debug1: identity file /home/devel/.ssh/id_xmss type -1
WARNING: ssh: debug1: identity file /home/devel/.ssh/id_xmss-cert type -1
WARNING: ssh: debug1: Local version string SSH-2.0-OpenSSH_8.4p1 Ubuntu-6ubuntu2.1
WARNING: ssh: debug1: Remote protocol version 2.0, remote software version OpenSSH_7.1
WARNING: ssh: debug1: match: OpenSSH_7.1 pat OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.4*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7* compat 0x04000002
WARNING: ssh: debug1: Authenticating to 169.254.21.34:22 as 'root'
WARNING: ssh: debug1: SSH2_MSG_KEXINIT sent
WARNING: ssh: debug1: SSH2_MSG_KEXINIT received
WARNING: ssh: debug1: kex: algorithm: curve25519-sha256@libssh.org
WARNING: ssh: debug1: kex: host key algorithm: ecdsa-sha2-nistp256
WARNING: ssh: debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
WARNING: ssh: debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
WARNING: ssh: debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
WARNING: ssh: debug1: Server host key: ecdsa-sha2-nistp256 SHA256:o6d6Bgm/lbi0vNqqY3V2uya9GAOmN/5QHLr5CT4iKlA
WARNING: ssh: Warning: Permanently added '169.254.21.34' (ECDSA) to the list of known hosts.
WARNING: ssh: debug1: rekey out after 134217728 blocks
WARNING: ssh: debug1: SSH2_MSG_NEWKEYS sent
WARNING: ssh: debug1: expecting SSH2_MSG_NEWKEYS
WARNING: ssh: debug1: SSH2_MSG_NEWKEYS received
WARNING: ssh: debug1: rekey in after 134217728 blocks
WARNING: ssh: debug1: Will attempt key: /home/devel/.ssh/id_rsa RSA SHA256:5KzmBzkhR/M1m3XEZ8npxZxgPRWGn8YTeKbhPE9bA34
WARNING: ssh: debug1: Will attempt key: /home/devel/.ssh/id_dsa
WARNING: ssh: debug1: Will attempt key: /home/devel/.ssh/id_ecdsa
WARNING: ssh: debug1: Will attempt key: /home/devel/.ssh/id_ecdsa_sk
WARNING: ssh: debug1: Will attempt key: /home/devel/.ssh/id_ed25519
WARNING: ssh: debug1: Will attempt key: /home/devel/.ssh/id_ed25519_sk
WARNING: ssh: debug1: Will attempt key: /home/devel/.ssh/id_xmss
WARNING: ssh: debug1: SSH2_MSG_SERVICE_ACCEPT received
WARNING: ssh: debug1: Authentications that can continue: publickey,password
WARNING: ssh: debug1: Next authentication method: publickey
WARNING: ssh: debug1: Offering public key: /home/devel/.ssh/id_rsa RSA SHA256:5KzmBzkhR/M1m3XEZ8npxZxgPRWGn8YTeKbhPE9bA34
WARNING: ssh: Received disconnect from UNKNOWN port 65535:2: Too many authentication failures
WARNING: ssh: Disconnected from UNKNOWN port 65535

[Bastian-Krause]

SSH server and client negotiate the authentication methods.

The target only supports password authentication.

Your SSH server still offers the authentication methods publickey, password. The negotiated authentication methods are tried in a defined order, see ssh_config(5) (PreferredAuthentications). "publickey" is preferred over "password". Combined with MaxAuthTries 1, the SSH client will never even try "password" authentication, unless you either limit PreferredAuthentications to "password" or reorder the authentication methods, so "password" is preferred over "publickey".

I guess your options are either to turn off "publickey" authentication in your sshd_config (if it's not used anyway) or teach labgrid's SSHDriver to set PreferredAuthentications based on self.keyfile/self.networkservice.password (basically what you suggested as a workaround, but also for the "publickey" case).

What's the reason to set MaxAuthTries anyway?

[NiklasReisser]

Hi,

First of all, sorry for the slow reply.

SSH server and client negotiate the authentication methods.

Yes - but how come they do it differenty depending on wether I go via labgrids proxy or not?

I guess your options are either to turn off "publickey" authentication in your sshd_config (if it's not used anyway) or teach labgrid's SSHDriver to set PreferredAuthentications based on self.keyfile/self.networkservice.password (basically what you suggested as a workaround, but also for the "publickey" case).

The latter is my current workaround - theoretically, would you accept this as PR? I'd prefer to not maintain my own labgrid version.

What's the reason to set MaxAuthTries anyway?

Good question. To be completely honest, I found it like this when I joined the project and did not ask that question yet.

[Bastian-Krause]

SSH server and client negotiate the authentication methods.

Yes - but how come they do it differenty depending on wether I go via labgrids proxy or not?

I don't understand that either. If you use the SSH client directly, you don't experience these issues? Judging from your configuration, I would have thought that you hit the exact same problem. Or do you use any special SSH command line flags or SSH client configuration there?

I guess your options are either to turn off "publickey" authentication in your sshd_config (if it's not used anyway) or teach labgrid's SSHDriver to set PreferredAuthentications based on self.keyfile/self.networkservice.password (basically what you suggested as a workaround, but also for the "publickey" case).

The latter is my current workaround - theoretically, would you accept this as PR? I'd prefer to not maintain my own labgrid version.

I'd like to leave this for @Emantor and @jluebbe to decide.

What's the reason to set MaxAuthTries anyway?

Good question. To be completely honest, I found it like this when I joined the project and did ask that question yet.

Might be worth an internal discussion. If there is no use case for pukey authentication and MaxAuthTries should stay, I think this is a misconfiguration.

[NiklasReisser]

If I connect via ssh directly from the exporter host, I get the same error as via proxy. This is what I would have expected from labgrid-client too, but connecting via labgrid-client directly (as in: On the same host as the exporter. The device is not reachable directly from anywhere else) works.

[Bastian-Krause]

Did you manage to find out what was going on here?

[NiklasReisser]

Nope, sorry. Also, I probably won't be looking into this anytime soon. It's on my to-do list, but reeeeally far down.