Can't connect anymore due to RSA drop in SSH 8.8

[cnieves1]

Hi I have autossh version 1.0.0 installed. I updated to HA OS 2022.6.0, and I'm not longer able to login to my remote server (which didn't changed). I get the following log:

debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug1: Connecting to xxx.xxx.xxx.xxx [xxx.xxx.xxx.xxx] port 22.
debug1: Connection established.
debug1: identity file /data/ssh_keys/autossh_rsa_key type 0
debug1: identity file /data/ssh_keys/autossh_rsa_key-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.8
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1
debug1: compat_banner: match: OpenSSH_6.6.1 pat OpenSSH_6.6.1* compat 0x04000002
debug1: Authenticating to xxx.xxx.xxx.xxx:22 as 'xxx'
debug1: load_hostkeys: fopen /root/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: SSH2_MSG_KEXINIT sent
autossh 1.4g
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-ed25519 SHA256:OoJ8yuspp/3ZXRyDt15uP4VatNRLuaVqLf4wQBUxMbE
debug1: load_hostkeys: fopen /root/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host 'xxx.xxx.xxx.xxx' is known and matches the ED25519 host key.
debug1: Found key in /root/.ssh/known_hosts:1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /data/ssh_keys/autossh_rsa_key RSA SHA256:9cMPtntARkdzSoQghRLUfXlAXPFIBryhMJXEs4e1Mu4 explicit
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: Next authentication method: publickey
debug1: Offering public key: /data/ssh_keys/autossh_rsa_key RSA SHA256:9cMPtntARkdzSoQghRLUfXlAXPFIBryhMJXEs4e1Mu4 explicit
debug1: send_pubkey_test: no mutual signature algorithm
debug1: Next authentication method: password
debug1: read_passphrase: can't open /dev/tty: No such device or address
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
Permission denied, please try again.
debug1: read_passphrase: can't open /dev/tty: No such device or address
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
Permission denied, please try again.
debug1: read_passphrase: can't open /dev/tty: No such device or address
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password
debug1: No more authentication methods to try.
xxx@xxx.xxx.xxx.xxx: Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).
s6-rc: info: service legacy-services: stopping
s6-rc: info: service legacy-services successfully stopped
s6-rc: info: service legacy-cont-init: stopping
s6-rc: info: service legacy-cont-init successfully stopped
s6-rc: info: service fix-attrs: stopping
s6-rc: info: service fix-attrs successfully stopped
s6-rc: info: service s6rc-oneshot-runner: stopping
s6-rc: info: service s6rc-oneshot-runner successfully stopped

It seems RSA is no longer supported by installed SSH: https://www.linuxadictos.com/en/openssh-8-8-arrives-saying-goodbye-to-ssh-rsa-support-bug-fixes-and-more.html

I tried to generate ed25519 SSH keys adding "-t ed25519" to "other SSH options" in AutoSSH configuration but it keeps generating a RSA key...

[17:02:42] INFO: Deleting existing key pair due to set 'force_keygen'
[17:02:42] WARNING: Do not forget to unset 'force_keygen' in your add-on configuration
[17:02:42] INFO: No previous key pair found
Generating public/private rsa key pair.
Your identification has been saved in /data/ssh_keys/autossh_rsa_key
Your public key has been saved in /data/ssh_keys/autossh_rsa_key.pub
The key fingerprint is:
SHA256:oevdF7AikQHnJFBf2FhFQuR2gN5sbV47SAVo4xAjuWw hassio-setup-via-autossh
The key's randomart image is:
+---[RSA 4096]----+
|  .o=o=XB++.     |
|    .B*+=o  .    |
|   . ooO+o..     |
|    E +o=o= .    |
|   .  .oS+ = .   |
|      ... + +    |
|      .. .   o   |
|     . . .  .    |
|      . . ..     |
+----[SHA256]-----+
[17:02:48] INFO: The public key is:
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDj9h7Rd51zDICYaK/KK1+apKc3OohdCFXkxlrKobSQycUw9Qfztn9s9PsHJkcOIS8oGIuYJMsyajW7hOBUCf5AXkUTgOiV4mP6Nu/dHlxl5C5HvbU5el19ref68hKwJJksPKd/OYgVGjn9v3um4BuxpH9gqLXqK9bCpCOPULGXzwvmM7xonzJc8T8Seh+dXBi+LXq6vJKzPTdNJFf9zHINbUCpDeoG/HZCSbNLrjW2yT5bj+BPtsnXbhap6/9oOBztVboFMeT2lwJ48oPUdoAmAgZ/gvgc4q97f1hecXCUxOHp1HAWDMffOBV4n0YbzUv8ywLVJJlozcjay4rYlDeFBuaQUD3yLxy16Yg2f1tGcyCAEyHSaWEsIohHi4G/IIL3CWldTwhNNBBe03tSbNty6UcGm74nwLDjVSjwnVLkYQjOKZOPYTE0HFI+iS9xL3y/ZMxBa2SGNiC5r1YmPhwbdWfKXcpdmb1mgNFK31y+Lo+osEG+7oNDwLITOizEelKJSWd+lYbMLGPs4nyaUduisbxIJcsiBdlwIToQv5y+XfRk6/gXb64hC4YUHyKEOQIDzbbgfWvIRGJUG7Lsi3OdJQGWadbalB15h71tep24Z2RkknSUgStx7clUWQaFHopBO26o1vmwD79akYZdvah2iOZ1IJbD3V1bZ2YvpAbfCQ== hassio-setup-via-autossh
[17:02:48] WARNING: Add this key to '~/.ssh/authorized_keys' on your remote server now!
[17:02:48] WARNING: Please restart add-on when done. Exiting...

How can I solve this?

Thanks in advance!

[ThomDietrich]

Hey thanks for reporting! Sounds severe.

The other SSH options do not apply to the key generation: https://github.com/ThomDietrich/home-assistant-addons/blob/master/autossh/run.sh#L27=

I see no harm in switching the standard key type. I will commit an untested change now, Would you please be so kind and let me know if version 1.0.8 resolves the issue!?

[ThomDietrich]

I've pushed the change to a branch but not yet to master: https://github.com/ThomDietrich/home-assistant-addons/commit/7a4f395671e7debdddeb486978eebd8811a66502

I'm troubled by the implications of this. Every setup of this add-on will need to replace their keys after the update. They might loose temporary access to their setup. I need to leave in a few minutes and have to think about this later. Any thoughts?

[cnieves1]

Sure! I can check the key algorithm in the log, but I have to wait until tomorrow to upload the key to the server and check the connection...

I have the auto update checked on for this addon, so it should update automatically. If it doesn't, do you know how to force the update manually?

Thanks,

[ThomDietrich]

[ThomDietrich]

As I said, this is in another branch and won't pop up as an automatic update.

I was actually thinking about a warning in the update release notes but people like you, with automatic update, won't read these. Damn...

I am thinking of users who can't physically reach their instance. The add-on will stop working and you and I are to blame :D

[cnieves1]

Can't find the update because it is on a branch... Tried with a file editor, but I couldn't find the file... Maybe because it is on a docker? I tried a 'find / -iname "autossh"' in the ssh docker with no results, I think because it is a different docker... Any ideas?

I frequently update my HAOS. For instance, this remote access stopped working some weeks ago, but I only recently realized it lays at the roots of the ssh algorithm..

I think you can ask the HA folks to drop a note in their blog about this. At least I read it... And for the future, it seems a good idea that the user could change the algorithm in the configuration...

[cnieves1]

By the way, I don't see any risk at changing the default key generation algorithm..

For users with this addon installed, everything will be the same until they switch on the force keygen option. The only thing you have to do is to be able to support both filenames: autossh_rsa_key.pub and autossh_key.pub. Maybe use the latter if it exists, and the former in case it doesn't...

[ThomDietrich]

I am going on vacation today and wanted to avoid to break anything. I've just pushed the update and it seems to work without any further action 👍 Thanks! Cheers

[Cogitri]

Unfortunately I think this still happens with HAOS 2022.9.6 and Fedora 36 as host. On my Fedora 36 host I get the following errors logged:

Sep 29 23:12:32 v2202201153511174265 sshd[504544]: Unable to negotiate with port 15749: no matching host key type found. Their offer: sk-ssh-ed25519@openssh.com [preauth] Sep 29 23:12:32 v2202201153511174265 sshd[504542]: Unable to negotiate with port 15817: no matching host key type found. Their offer: sk-ecdsa-sha2-nistp256@openssh.com [preauth]

[ThomDietrich]

Hey @Cogitri, "Fedora 36 as host" you mean Fedora on the SSH server? Otherwise it seems to work quite well so far

[Cogitri]

Yup, seems like the Fedora 36/37 OpenSSH 8.8 server doesn't like the ed25519 key. Changing it to a (secure) RSA one works (I just changed this in my fork for now)

[ThomDietrich]

That's annoying. How do you recon we solve this? I would really hate to need to introduce this as a config option, but might be the only solution!?

[Cogitri]

Hm, possibly. I have to admit I'm not quite sure why the Fedora server doesn't accept the ED25519 key, so maybe a config option would work.

[ThomDietrich]

Could you do us a favor and do a bit of research? I can't imagine a modern system to not support this new key type, which is available and recommended for a long time. If still needed, happy to accept a pull-request with an optional config parameter, defaulting to ed25519.

@cnieves1 what do you think?