Cannot authenticate with a private key

[walac]

Describe the bug

When I try to connect to a host using a private key, it doesn't work. With the ssh command line client, it works fines.

To Reproduce

Steps to reproduce the behavior:

1. Run podman run -p 2022:22 docker.io/walac/pyktest-ssh:latest (the docker image is publicly available)
2. Run the following python script

   
   from pssh.clients import SSHClient
   import logging as l

l.getLogger('pssh.host_logger').setLevel(l.DEBUG) l.getLogger('pssh').setLevel(l.DEBUG)

pkey = \ b"""-----BEGIN OPENSSH PRIVATE KEY----- b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAABlwAAAAdzc2gtcn NhAAAAAwEAAQAAAYEApq7NTeQ4JsLMTdZdiV3UMUzvj8wcmZbsMozUnNXM6LK9fBxZkbx9 /rTZlgPMBHFhJZcTMAH4/gkWJZ4yBMCHu1oh9jyzPNbdSMcCKlJxMzmnTebGDbK7acBBiC hHOy2IzIbaM/M6bqk+OKpbCRx0rPNcwAx/YZq4E73+hZ3ZnODV0n7UAS2Z42MBFOW8/Jn7 +XyKQLozjn+iEs/BZ5SKU5d6WNpbqh/Dka5pG3HhtQDU3zq4nPn53mE2jYZly5it7dBPv9 C985FHPDXmeO+l6DNPFtpX+MdTW6Umf2ZBuSMai2p9idE/BsF69j4kuLBT8LgjT3sqX4Ph 1pA4zO1w2Uuszv6We6PlQ/BX4n+k1SPc+EiPljK9RxRB8pDCNuZI8yWlrloCCmyE7KSOkr Z0l7aArHs4i6fqL66bwD3ZZfHqfMV/8VwIQV//YQqYfugmWqsbFPeEJkoNb0pnUYfFiRy3 4qCsRz3pfxXeWhIoQsT8iLKvYvZDaNmrDIdjXw1TAAAFiHgCmvh4Apr4AAAAB3NzaC1yc2 EAAAGBAKauzU3kOCbCzE3WXYld1DFM74/MHJmW7DKM1JzVzOiyvXwcWZG8ff602ZYDzARx YSWXEzAB+P4JFiWeMgTAh7taIfY8szzW3UjHAipScTM5p03mxg2yu2nAQYgoRzstiMyG2j PzOm6pPjiqWwkcdKzzXMAMf2GauBO9/oWd2Zzg1dJ+1AEtmeNjARTlvPyZ+/l8ikC6M45/ ohLPwWeUilOXeljaW6ofw5GuaRtx4bUA1N86uJz5+d5hNo2GZcuYre3QT7/QvfORRzw15n jvpegzTxbaV/jHU1ulJn9mQbkjGotqfYnRPwbBevY+JLiwU/C4I097Kl+D4daQOMztcNlL rM7+lnuj5UPwV+J/pNUj3PhIj5YyvUcUQfKQwjbmSPMlpa5aAgpshOykjpK2dJe2gKx7OI un6i+um8A92WXx6nzFf/FcCEFf/2EKmH7oJlqrGxT3hCZKDW9KZ1GHxYkct+KgrEc96X8V 3loSKELE/Iiyr2L2Q2jZqwyHY18NUwAAAAMBAAEAAAGANzoy3yyPaFRh4iZ1TobuwDkIJS KRlVg6wZME/UQfxTg37U/tY4rLSmH8uCZg6lXwxMY2PtFggTdchbFRTF7IekymdRJupulg X3VE4+X2CO2A4CBnHfHLBAKGFAmYdGSlb3L9CHp+MV1VNuxHBpFNnWJzKFTbvejINg5dL0 N6b27Vmg2YzB0iYIOUv6pRQHb61JKtsj2fyIbEmiwJHntKi+DkUl1jQs6Me5AI1mWYzeLN JXus4KI45rRaIXFW+CsJaFotgbRdXLcS23nehPhdzzD+vLlvb3uHIxtV4vFnjpVthH94Ov Sr6jNXp+U9r6GqO9PrZe5FImFkA1ACd9HbkTCaptAoRorq/EG/BeDH7sZ+HXb+byxBzdXL rbE4CTOSGVjssEX/sxj58KsOMaiA+h3+DFLA2qqbYD/+gCwK+9E+rBAU1zeafWvSxHrPzr rvFz01u8CdqilXFYeCRDsj3xkuQPlRimHrX6w/X/qvZ7pSQTr+apIU3O+V81FTvSbNAAAA wQCtjd+HU8pwwXlNH3gEGX+YIQEgmRwgRXQm3mONEGsY/2udPfqtuPzNpTcqcStc+hhYlA ndFQLKIPJktNOIrc8D2d/VdvXUX8YGiCDQNWOOCmXcMK3frp8Q7NWFWGDCBUZictbpusDe 5ulROhtD2T5T4FC9inJLK1cfIqV/Gu36eoH6aBKRzTsub1rM0VBFqo7lVgKytK2Q60MnR8 TZe1au5gTV17oULelmYVj+mGiaojc9lPt5MtaUZlQFjbVxpaQAAADBALnFkNIrMbnhHk+J J66VlKvr2Bm+ihzwgbOxYDYKqPL+rasLWuiY0xmEbmIgV89ZXRaOwYJWoMpaVuTue7+swx viF97lOBAAPpmk8MZcBF31n8Pk8LTlOuzH2PR/t5t2hJsHazM/+nFAto2mccvnMkpUAg8X WYltoLPf+fNqcpshx5TEMYLR58PQ4xVHJZQmY3Ysqy1si6kRuvYfArgk3VKadtDbXORk0g Hq6Hl748Z/cd+w8C0j+v+ynPK0+7OBtQAAAMEA5bHeFUQtvRTOBCUMwV6jZ7alKj9+mZDR eMQCkvSQTbDLSVgeL6GPg69xZfrfnea/tSKnTqibYH43FXKVaX0ePsNjmzZLAEPx2AjWNh uHnmqem9tuYG/bldVj+GcqeAanyRPY9PvWY27jG1nq8kTMG03NZLoXKxKG5WToc6aMDdzA R13A9j3abQ/gp+n/KCWpGGwm8845WdWNzgc4QCvnEot1H53STHXjHpkaGnLM9ERW3nHuxg 1syV87ktSaXtfnAAAADXdjb3N0YUBmZWRvcmEBAgMEBQ== -----END OPENSSH PRIVATE KEY-----"""

ssh = SSHClient(host='localhost', user='root', pkey=pkey, port=2022, num_retries=1, identity_auth=False)

**Expected behavior**

A connection to the remote host is established.

**Actual behavior**

Traceback (most recent call last): File "/home/wcosta/.local/lib/python3.10/site-packages/pssh/clients/base/single.py", line 204, in _auth_retry self.auth() File "/home/wcosta/.local/lib/python3.10/site-packages/pssh/clients/base/single.py", line 349, in auth return self._pkey_auth(self.pkey) File "/home/wcosta/.local/lib/python3.10/site-packages/pssh/clients/base/single.py", line 389, in _pkey_auth return self._pkey_from_memory(_pkey) File "/home/wcosta/.local/lib/python3.10/site-packages/pssh/clients/native/single.py", line 255, in _pkey_from_memory self.session.userauth_publickey_frommemory( File "ssh2/session.pyx", line 293, in ssh2.session.Session.userauth_publickey_frommemory File "ssh2/utils.pyx", line 166, in ssh2.utils.handle_error_codes ssh2.exceptions.AuthenticationError

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/wcosta/test.py", line 47, in ssh = SSHClient(host='localhost', File "/home/wcosta/.local/lib/python3.10/site-packages/pssh/clients/native/single.py", line 138, in init super(SSHClient, self).init( File "/home/wcosta/.local/lib/python3.10/site-packages/pssh/clients/base/single.py", line 188, in init self._init() File "/home/wcosta/.local/lib/python3.10/site-packages/pssh/clients/base/single.py", line 196, in _init self._auth_retry() File "/home/wcosta/.local/lib/python3.10/site-packages/pssh/clients/base/single.py", line 210, in _auth_retry raise AuthenticationError(msg, self.host, self.port, ex, retries, self.num_retries) pssh.exceptions.AuthenticationError: ('Authentication error while connecting to %s:%s - %s - retries %s/%s', 'localhost', 2022, AuthenticationError(), 1, 1)

**Additional information**

parallel-ssh in ./.local/lib/python3.10/site-packages (2.12.0) ssh-python in /usr/lib64/python3.10/site-packages (from parallel-ssh) (0+unknown) gevent>=1.3.0 in ./.local/lib/python3.10/site-packages (from parallel-ssh) (21.12.0) ssh2-python in ./.local/lib/python3.10/site-packages (from parallel-ssh) (1.0.0) zope.event in ./.local/lib/python3.10/site-packages (from gevent>=1.3.0->parallel-ssh) (4.5.0) zope.interface in ./.local/lib/python3.10/site-packages (from gevent>=1.3.0->parallel-ssh) (5.4.0) setuptools in /usr/lib/python3.10/site-packages (from gevent>=1.3.0->parallel-ssh) (59.6.0) greenlet<2.0,>=1.1.0 in /usr/lib64/python3.10/site-packages (from gevent>=1.3.0->parallel-ssh) (1.1.2)

[V346Cen]

@walac what is the remote SSH server you're trying to connect to, and what version? I've seen a problem that's very similar to yours. In my testing with parallel-ssh-2.12.0 I had no trouble connecting to OpenSSH_8.5p1, but I couldn't connect to OpenSSH_8.8p1 server.

Here's my example code and the result when trying to connect with OpenSSH_8.8p1: https://gist.github.com/V346Cen/367f9fc20cfa471567ac6f71f5efa51b. The key I used for testing was SSH-RSA, but I got the same results with ED25519 key. Unfortunately the debug prints don't provide much information about the problem.

I was able to "fix" my example by changing the import statement in line 4:

-from pssh.clients import ParallelSSHClient
+from pssh.clients.ssh.parallel import ParallelSSHClient

My understanding is that this changes the library internally used by parallel-ssh from ssh2-python to ssh-python. After this change my example works with both remote OpenSSH versions.

I suppose the problem could be somewhere further down parallel-ssh dependency chain. OpenSSH has seen some breaking changes in recent versions, they dropped some cryptographic routines considered unsafe nowadays. So maybe an adaptation is required somewhere?

Edit: Please disregard the part about ED25519 key, it was an error in my testing.

[walac]

@V346Cen my SSH server is OpenSSH_8.8p1, OpenSSL 3.0.5 5 Jul 2022, so I guess I am having the same problem of yours.

[pkittenis]

Hi,

This sounds like a compatibility issue with OpenSSH 8.8 and libssh2.

Please try the clients under from pssh.clients.ssh import ParallelSSHClient, SSHClient - likely they will work given the above.

To resolve the issue will need a newer libssh2 and ssh2-python - leaving this open to track.

[V346Cen]

I had to investigate this further, so here's the outcome. Maybe it will be useful to someone who finds this thread before the problem is fixed.

The problem is related to ssh-rsa public key signature algorithm. It is based on SHA-1 hashing function and is considered insecure. Starting from OpenSSH 8.8 this algorithm is disabled by default in SSH server. [1]

If SSH client connecting to such server is unable to use newer public key signature algorithms based on SHA-2, then it will not be able to connect using ssh-rsa keys. Apparently ssh2-python is affected, while ssh-python is not.

There are three possible workarounds at this point:

1. Use ssh-python backend.
2. Use a different type of SSH keys instead of ssh-rsa (e.g. ed25519).
3. Explicitly re-enable ssh-rsa public key signature algorithm on SSH server side (not recommended).

[1] https://www.openssh.com/txt/release-8.8

[rmarops]

Option 3 does not work me against Amazon Linux 2023 and pssh using a SHA-256 rsa key. Same key used outside of pssh from command line works fine. For the record, I tried PubkeyAcceptedAlgorithms +ssh-rsa , PubkeyAcceptedKeyTypes +ssh-rsa, HostKeyAlgorithms +ssh-rsa while trying to get it to work for the short term. Debug always shows: userauth_pubkey: key type ssh-rsa not in PubkeyAcceptedAlgorithms [preauth]. Seems hard disabled.

[sjackson-adaptiv]

Please try the clients under from pssh.clients.ssh import ParallelSSHClient, SSHClient - likely they will work given the above.

I want to add that this worked for me, thanks for the suggestion!

[max-l]

I have the same problem, with both "rsa" and "ed25519" keys, see the code below.

Both of the keys DO work on the same server (and same account), with the ssh terminal client.

If I log in from the terminal with the verbose flag (-v), I get the following server side info:

debug1: Local version string SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.6
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4
debug1: compat_banner: match: OpenSSH_7.4 pat OpenSSH_7.4* compat 0x04000006

Interestingly the same test code below DOES work with another server, that has the following ssh server side info:

debug1: Local version string SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.6
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.0
debug1: compat_banner: match: OpenSSH_8.0 pat OpenSSH* compat 0x04000000

import os
from pssh.clients import SSHClient

def test_pssh():

    host = " ... my host ..."

    pkey = os.path.expanduser("~/.ssh/id_ed25519")

    ssh_client = SSHClient(host, "my-username", pkey=pkey)

    cmd = "ls -al"

    host_out = ssh_client.run_command(cmd)
    stdout_txt = "\n".join(list(host_out.stdout))
    stderr_lines = list(host_out.stderr)

    if host_out.exit_code != 0:
        stderr_txt = "\n".join(stderr_lines)
        raise Exception(f"remote call failed {host_out.exit_code}, '{cmd}'\n{stderr_txt}\n")

    print(stdout_txt)

Traceback (most recent call last):
  File "/home/maxl/dev/DryPipe/venv/lib/python3.10/site-packages/parallel_ssh-2.12.0-py3.10.egg/pssh/clients/base/single.py", line 204, in _auth_retry
    self.auth()
  File "/home/maxl/dev/DryPipe/venv/lib/python3.10/site-packages/parallel_ssh-2.12.0-py3.10.egg/pssh/clients/base/single.py", line 349, in auth
    return self._pkey_auth(self.pkey)
  File "/home/maxl/dev/DryPipe/venv/lib/python3.10/site-packages/parallel_ssh-2.12.0-py3.10.egg/pssh/clients/base/single.py", line 389, in _pkey_auth
    return self._pkey_from_memory(_pkey)
  File "/home/maxl/dev/DryPipe/venv/lib/python3.10/site-packages/parallel_ssh-2.12.0-py3.10.egg/pssh/clients/native/single.py", line 255, in _pkey_from_memory
    self.session.userauth_publickey_frommemory(
  File "ssh2/session.pyx", line 293, in ssh2.session.Session.userauth_publickey_frommemory
  File "ssh2/utils.pyx", line 168, in ssh2.utils.handle_error_codes
ssh2.exceptions.PublickeyUnverifiedError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/maxl/dev/DryPipe/venv/lib/python3.10/site-packages/parallel_ssh-2.12.0-py3.10.egg/pssh/clients/base/single.py", line 204, in _auth_retry
    self.auth()
  File "/home/maxl/dev/DryPipe/venv/lib/python3.10/site-packages/parallel_ssh-2.12.0-py3.10.egg/pssh/clients/base/single.py", line 349, in auth
    return self._pkey_auth(self.pkey)
  File "/home/maxl/dev/DryPipe/venv/lib/python3.10/site-packages/parallel_ssh-2.12.0-py3.10.egg/pssh/clients/base/single.py", line 389, in _pkey_auth
    return self._pkey_from_memory(_pkey)
  File "/home/maxl/dev/DryPipe/venv/lib/python3.10/site-packages/parallel_ssh-2.12.0-py3.10.egg/pssh/clients/native/single.py", line 255, in _pkey_from_memory
    self.session.userauth_publickey_frommemory(
  File "ssh2/session.pyx", line 293, in ssh2.session.Session.userauth_publickey_frommemory
  File "ssh2/utils.pyx", line 166, in ssh2.utils.handle_error_codes
ssh2.exceptions.AuthenticationError

[comperem]

hello @pkittenis , thank you, this worked for me.

the different approach to loading ParallelSSHClient (using ssh-python) brought success:

from pssh.clients.ssh.parallel import ParallelSSHClient