gravitational / teleport

The easiest, and most secure way to access and protect all of your infrastructure.
https://goteleport.com
GNU Affero General Public License v3.0
17.41k stars 1.74k forks source link

`tctl auth sign --format=openssh` generates a `ssh-rsa-cert-v01` public key breaking centos7 to alma9 ssh #40100

Open amarinderca opened 6 months ago

amarinderca commented 6 months ago

Expected behavior:

As mentioned in #10918 ssh-rsa-cert-v01 is no longer on the approval list of PubKeyAcceptedAlgorithms for many new OSes like Alma 9.

This issue of teleport generating ssh-rsa-cert-v01 public keys still present with tctl auth sign --format=openssh --host=myhost --out=myhost when manually enrolling agentless hosts.

This specifically causes issues when connecting from centos7 -> alma9 machines directly as the ssh command fails.

We are able to ssh from alma9 machines to centos7 machines.

Current behavior:

CASignatureAlgorithms=+ssh-rsa is not a valid option on centos 7 with ssh version: OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017

[a@admin ~]$ ssh -A teleport -o CASignatureAlgorithms=+ssh-rsa -o PubkeyAcceptedAlgorithms=+ssh-rsa-cert-v01@openssh.com
command-line: line 0: Bad configuration option: casignaturealgorithms

[a@admin ~]$ ssh -A teleport -o PubkeyAcceptedAlgorithms=+ssh-rsa-cert-v01@openssh.com
command-line: line 0: Bad configuration option: pubkeyacceptedalgorithms
[a@admin ~]$ ssh -A teleport -o PubkeyAcceptedKeyTypes=+ssh-rsa-cert-v01@openssh.com
Connection closed by 10.4.30.165 port 22

We are not able to ssh from centos7 machines to alma9 machines.

Bug details:

/var/log/messages

Apr  1 22:47:46 p1.hc.domain.com sshd[470936]: fatal: mm_answer_sign: sign: error in libcrypto
Apr  1 22:48:31 p1.hc.domain.com sshd[470948]: fatal: mm_answer_sign: sign: error in libcrypto
Apr  1 22:48:42 p1.hc.domain.com sshd[470951]: fatal: mm_answer_sign: sign: error in libcrypto
Apr  1 22:49:31 p1.hc.domain.com sshd[471071]: fatal: mm_answer_sign: sign: error in libcrypto
Apr  1 22:50:11 p1.hc.domain.com sshd[471145]: fatal: mm_answer_sign: sign: error in libcrypto
[a@tester ~]$ ssh -vvv p1.hc.domain.com
OpenSSH_7.4p1, OpenSSL 1.0.2k-fips  26 Jan 2017
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 58: Applying options for *
debug2: resolving "p1.hc.domain.com" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to p1.hc.domain.com [10.4.30.114] port 22.
debug1: Connection established.
debug1: key_load_public: No such file or directory
debug1: identity file /home/a/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/a/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/a/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/a/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/a/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/a/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/a/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/a/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.4
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.7
debug1: match: OpenSSH_8.7 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to p1.hc.domain.com:22 as 'a'
debug3: hostkeys_foreach: reading file "/home/a/.ssh/known_hosts"
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ssh-dss
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,kex-strict-s-v00@openssh.com
debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com
debug2: ciphers ctos: aes256-ctr,aes192-ctr,aes128-ctr
debug2: ciphers stoc: aes256-ctr,aes192-ctr,aes128-ctr
debug2: MACs ctos: hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com
debug2: MACs stoc: hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ssh-rsa-cert-v01@openssh.com
debug1: kex: server->client cipher: aes128-ctr MAC: umac-128-etm@openssh.com compression: none
debug1: kex: client->server cipher: aes128-ctr MAC: umac-128-etm@openssh.com compression: none
debug1: kex: curve25519-sha256 need=16 dh_need=16
debug1: kex: curve25519-sha256 need=16 dh_need=16
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
Connection closed by 10.4.30.114 port 22
amarinderca commented 5 months ago

The "workaround" is to:

doabu commented 5 months ago

Any plans to upgrade the tctl auth sign command to allow create 'better' than the current type: ssh-rsa-cert-v01@openssh.com host certificate? I tried quite a bunch of workarounds found here (even not preferred once like reenabling legacy stuff), but still having issues with native opensshd on Alma Linux.

amarinderca commented 5 months ago

@doabu Take a look at allowing ssh-rsa-cert-v01@openssh.com for OpenSSH only from the crypto policy from rhel 9 docs here: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/security_hardening/using-the-system-wide-cryptographic-policies_security-hardening#excluding-an-application-from-following-the-system-wide-crypto-policies_using-the-system-wide-cryptographic-policies

For Alma 9: PubkeyAcceptedAlgorithms and HostKeyAlgorithms for both server and client need an override. You can copy the current crypto policy from /etc/ssh/sshd_config.d/50-redhat.conf and append ssh-rsa-cert-v01@openssh.com to PubkeyAcceptedAlgorithms and HostKeyAlgorithms in /etc/ssh/sshd_config.d/49-crypto-policy-override.conf

For CentOS 7: I have ensured that all "weak" Algos are disabled in both server /etc/ssh/sshd_config and client /etc/ssh/ssh_config config. You can get a full list of weak algos from many places. Rule of thumb seems to be to disable anything sha1. However, make sure to test this if you have any legacy systems with older OS that only accept weak algos and make appropriate exceptions.

Hope this helps. I can write up a longer post with extract settings that work for me.

dansou901 commented 3 months ago

I've got a similar problem. Just installed a completely new teleport cluster on a newly fresh installed Ubuntu VM (24.04) into microk8s. Then enrolled an SSH Host there via token create and teleport join. After that, tried to login to that Host via Teleport Web UI and got in the SSH Logs of the Host

userauth_pubkey: signature algorithm ssh-rsa-cert-v01@openssh.com not in PubkeyAcceptedAlgorithms [preauth]

which indicates that teleport still uses the old signature algorithm. When doing a tsh login from a different machine, I could verify with ssh-add -L that the certificates are indeed signed with ssh-rsa-cert-v01. Is there a possibility to configure that behaviour somehow? I haven't found it yet and I don't want to allow the old signature algorithm in our systems.