Unable to connect to the server (anymore)

[kszys]

I was using this addon to have remote access to my HA installation - it was working great! Until around 5th of July 2022 - probably linked to the update of the OpenSSH to version 9 (maybe?).

Right now the addon is not able to connect to the server anymore. The weird thing is that the connection is actually established and then disconnected. The best of my understanding the server claims that the user disconnected, and the client says that it is the server fault... Lost now... Here are the logs:

Server logs:

Jul  8 21:34:37 balder sshd[66272]: Unable to negotiate with 109.133.143.15 port 36230: no matching host key type found. Their offer: sk-ecdsa-sha2-nistp256@openssh.com [preauth]
Jul  8 21:34:37 balder sshd[66273]: Unable to negotiate with 109.133.143.15 port 36232: no matching host key type found. Their offer: sk-ssh-ed25519@openssh.com [preauth]
Jul  8 21:34:37 balder sshd[66279]: Accepted publickey for ha from 109.133.143.15 port 36234 ssh2: ED25519 SHA256:45mBPrf/4Jej6tLiVRp3J5y1UdkGdRMmGISHoRqE1Us
Jul  8 21:34:37 balder nologin[66282]: Attempted login by ha on /dev/pts/3
Jul  8 21:34:37 balder sshd[66281]: Received disconnect from 109.133.143.15 port 36234:11: disconnected by user
Jul  8 21:34:37 balder sshd[66281]: Disconnected from user ha 109.133.143.15 port 36234

Client (autossh) logs:

[23:35:26] INFO: Executing command: /usr/bin/autossh  -M 0 -o ServerAliveInterval=30 -o ServerAliveCountMax=3 -o StrictHostKeyChecking=no -o ExitOnForwardFailure=yes -p 22 -t -t -i /data/ssh_keys/autossh_rsa_key ha@95.179.134.83 -R 127.0.0.1:8123:172.17.0.1:8123 -v
autossh 1.4g
OpenSSH_9.0p1, OpenSSL 1.1.1o  3 May 2022
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Connecting to 95.179.134.83 [95.179.134.83] port 22.
debug1: Connection established.
debug1: identity file /data/ssh_keys/autossh_rsa_key type 3
debug1: identity file /data/ssh_keys/autossh_rsa_key-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_9.0
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.9 FreeBSD-20200214
debug1: compat_banner: match: OpenSSH_7.9 FreeBSD-20200214 pat OpenSSH* compat 0x04000000
debug1: Authenticating to 95.179.134.83:22 as 'ha'
debug1: load_hostkeys: fopen /root/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ssh-ed25519
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-ed25519 SHA256:s4XDZ7q60usITEoOSOa6b+XSqUVU5a8BgPxThCLejN8
debug1: load_hostkeys: fopen /root/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host '95.179.134.83' is known and matches the ED25519 host key.
debug1: Found key in /root/.ssh/known_hosts:1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /data/ssh_keys/autossh_rsa_key ED25519 SHA256:45mBPrf/4Jej6tLiVRp3J5y1UdkGdRMmGISHoRqE1Us explicit
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,keyboard-interactive
debug1: Next authentication method: publickey
debug1: Offering public key: /data/ssh_keys/autossh_rsa_key ED25519 SHA256:45mBPrf/4Jej6tLiVRp3J5y1UdkGdRMmGISHoRqE1Us explicit
debug1: Server accepts key: /data/ssh_keys/autossh_rsa_key ED25519 SHA256:45mBPrf/4Jej6tLiVRp3J5y1UdkGdRMmGISHoRqE1Us explicit
Authenticated to 95.179.134.83 ([95.179.134.83]:22) using "publickey".
debug1: Remote connections from 127.0.0.1:8123 forwarded to local address 172.17.0.1:8123
debug1: ssh_init_forwarding: expecting replies for 1 forwards
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: filesystem
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug1: client_input_hostkeys: searching /root/.ssh/known_hosts for 95.179.134.83 / (none)
debug1: client_input_hostkeys: searching /root/.ssh/known_hosts2 for 95.179.134.83 / (none)
debug1: client_input_hostkeys: hostkeys file /root/.ssh/known_hosts2 does not exist
debug1: Remote: /usr/home/ha/.ssh/authorized_keys:2: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug1: Remote: /usr/home/ha/.ssh/authorized_keys:2: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug1: remote forward success for: listen 127.0.0.1:8123, connect 172.17.0.1:8123
debug1: forwarding_success: all expected forwarding replies received
debug1: ssh_tty_make_modes: no fd or tio
debug1: client_global_hostkeys_private_confirm: server used untrusted RSA signature algorithm ssh-rsa for key 0, disregarding
Learned new hostkey: ECDSA SHA256:XycGYDpcQ25tjPJUW5IQ+NfG4FhIsFTkmilMP2yUHZg
Adding new key for 95.179.134.83 to /root/.ssh/known_hosts: ecdsa-sha2-nistp256 SHA256:XycGYDpcQ25tjPJUW5IQ+NfG4FhIsFTkmilMP2yUHZg
debug1: update_known_hosts: known hosts file /root/.ssh/known_hosts2 does not exist
Last login: Fri Jul  8 21:34:37 2022 from 15.143-133-109.adsl-dyn.isp.belgacom.be

FreeBSD 12.3-RELEASE-p5 GENERIC 

Welcome to FreeBSD!

Release Notes, Errata: https://www.FreeBSD.org/releases/
Security Advisories:   https://www.FreeBSD.org/security/
FreeBSD Handbook:      https://www.FreeBSD.org/handbook/
FreeBSD FAQ:           https://www.FreeBSD.org/faq/
Questions List: https://lists.FreeBSD.org/mailman/listinfo/freebsd-questions/
FreeBSD Forums:        https://forums.FreeBSD.org/

Documents installed with the system are in the /usr/local/share/doc/freebsd/
directory, or can be installed later with:  pkg install en-freebsd-doc
For other languages, replace "en" with a language code like de or fr.

Show the version of FreeBSD installed:  freebsd-version ; uname -a
Please include that output and any error messages when posting questions.
Introduction to manual pages:  man man
FreeBSD directory layout:      man hier

Edit /etc/motd to change this login announcement.
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: client_input_channel_req: channel 0 rtype eow@openssh.com reply 0
This account is currently not available.
debug1: channel 0: free: client-session, nchannels 1
Connection to 95.179.134.83 closed.
Transferred: sent 2708, received 4128 bytes, in 0.1 seconds
Bytes per second: sent 47946.6, received 73088.5
debug1: Exit status 1
s6-rc: info: service legacy-services: stopping
s6-rc: info: service legacy-services successfully stopped
s6-rc: info: service legacy-cont-init: stopping
s6-rc: info: service legacy-cont-init successfully stopped
s6-rc: info: service fix-attrs: stopping
s6-rc: info: service fix-attrs successfully stopped
s6-rc: info: service s6rc-oneshot-runner: stopping
s6-rc: info: service s6rc-oneshot-runner successfully stopped

Any idea how to solve it?

[kszys]

So... mystery solved, though I still have no clue why it was working before, and not any longer, as I have not made any changes on the server when it stopped working...

Anyway - the clue is in the statement: This account is currently not available. This message is apparently provided by the nologin shell. So it was sufficient to change the default shell for ha user to bash and magically everything works...

After some thinking, I decided that ha user should not really need a login shell on my server. So I changed the default shell back to nologin and added instead option -N in the additional options in autossh configuration - and it still works!

Again - why it worked before, but no any longer - no idea.

[ThomDietrich]

Hey! I am sorry, this might have been caused by a change here... https://github.com/ThomDietrich/home-assistant-addons/issues/1 The user needed the login shell to see relevant information. That's why I moved the -N from default to "additonal options", forgetting that users like you might be affected.

Btw your case sounds like you would benefit from he Docker based SSH daemon on the server, which I've described in DOCS.md

[kszys]

You are probably right. I got to the bottom of it, so no issues. But perhaps it would be useful to document it a bit more explicitly - either use the -N option, or ensure the user has a shell defined.

For Docker - you are probably right, but I do not use Docker on my server, and I was too lazy to set it up just for this ;)

[ThomDietrich]

The -N is part of the default options now: https://github.com/ThomDietrich/home-assistant-addons/commit/07316f35dc307cf7020451dc21ef806c5c99ea29 Of course existing users won't have that in their setup after a regular update. I will investigate ho to add a document with release notes for a home assistant addon and provide the info through that. It's not ideal but every user with an issue like yours will probably have a look at the version changes first. Agreed?

[ThomDietrich]

I'm happy with that :)

[kszys]

Looks good! Thanks!