gravitational / teleport

The easiest, and most secure way to access and protect all of your infrastructure.
https://goteleport.com
GNU Affero General Public License v3.0
17.58k stars 1.76k forks source link

Certificate presented by root proxy for leaf agentless nodes is not trusted by client #36801

Open strideynet opened 9 months ago

strideynet commented 9 months ago

Expected behavior: When connecting to an agentless node in a leaf cluster, no TOFU dialogue is presented

Current behavior:

➜  teleport git:(branch/v15) ✗ build/tsh status
> Profile URL:        [https://root.tele.ottr.sh:443](https://root.tele.ottr.sh/)
  Logged in as:       noah
  Cluster:            root.tele.ottr.sh
  Roles:              access, auditor, editor
  Logins:             root, noah
  Kubernetes:         enabled
  Valid until:        2024-01-17 22:22:53 +0000 GMT [valid for 12h0m0s]
  Extensions:         login-ip, permit-agent-forwarding, permit-port-forwarding, permit-pty, private-key-policy
# Begin generated Teleport configuration for root.tele.ottr.sh by tsh

# Common flags for all root.tele.ottr.sh hosts
Host *.root.tele.ottr.sh root.tele.ottr.sh
    UserKnownHostsFile "/Users/noah/.tsh/known_hosts"
    IdentityFile "/Users/noah/.tsh/keys/root.tele.ottr.sh/noah"
    CertificateFile "/Users/noah/.tsh/keys/root.tele.ottr.sh/noah-ssh/root.tele.ottr.sh-cert.pub"
    HostKeyAlgorithms rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com

# Flags for all root.tele.ottr.sh hosts except the proxy
Host *.root.tele.ottr.sh !root.tele.ottr.sh
    Port 3022
    ProxyCommand "/Users/noah/code/gravitational/teleport/build/tsh" proxy ssh --cluster=root.tele.ottr.sh --proxy=root.tele.ottr.sh:443 %r@%h:%p
# Common flags for all leaf.tele.ottr.sh hosts
Host *.leaf.tele.ottr.sh root.tele.ottr.sh
    UserKnownHostsFile "/Users/noah/.tsh/known_hosts"
    IdentityFile "/Users/noah/.tsh/keys/root.tele.ottr.sh/noah"
    CertificateFile "/Users/noah/.tsh/keys/root.tele.ottr.sh/noah-ssh/root.tele.ottr.sh-cert.pub"
    HostKeyAlgorithms rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com

# Flags for all leaf.tele.ottr.sh hosts except the proxy
Host *.leaf.tele.ottr.sh !root.tele.ottr.sh
    Port 3022
    ProxyCommand "/Users/noah/code/gravitational/teleport/build/tsh" proxy ssh --cluster=leaf.tele.ottr.sh --proxy=root.tele.ottr.sh:443 %r@%h:%p

# End generated Teleport configuration
➜  teleport git:(branch/v15) ✗ ssh -p 22 -F ssh_config noah@noah-v15-agentless.leaf.tele.ottr.sh -vv ls
OpenSSH_9.4p1, LibreSSL 3.3.6
debug1: Reading configuration data ssh_config
debug1: ssh_config line 15: Applying options for *.leaf.tele.ottr.sh
debug1: ssh_config line 22: Applying options for *.leaf.tele.ottr.sh
debug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling
debug1: Executing proxy command: exec "/Users/noah/code/gravitational/teleport/build/tsh" proxy ssh --cluster=leaf.tele.ottr.sh --proxy=root.tele.ottr.sh:443 noah@noah-v15-agentless.leaf.tele.ottr.sh:22
debug1: identity file /Users/noah/.tsh/keys/root.tele.ottr.sh/noah type 0
debug1: certificate file /Users/noah/.tsh/keys/root.tele.ottr.sh/noah-ssh/root.tele.ottr.sh-cert.pub type 4
debug1: Local version string SSH-2.0-OpenSSH_9.4
debug1: Remote protocol version 2.0, remote software version Go
debug1: compat_banner: no match: Go
debug2: fd 5 setting O_NONBLOCK
debug2: fd 4 setting O_NONBLOCK
debug1: Authenticating to noah-v15-agentless.leaf.tele.ottr.sh:22 as 'noah'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: sntrup761x25519-sha512@openssh.com,curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c
debug2: host key algorithms: rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group14-sha256,kex-strict-s-v00@openssh.com
debug2: host key algorithms: rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com
debug2: ciphers ctos: aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr
debug2: ciphers stoc: aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr
debug2: MACs ctos: hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha2-512
debug2: MACs stoc: hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha2-512
debug2: compression ctos: none
debug2: compression stoc: none
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: rsa-sha2-512-cert-v01@openssh.com
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host certificate: ssh-rsa-cert-v01@openssh.com SHA256:Cwo2MauQbTIO8kH3s3pDAHYUFK8cCXwLvkjCK1r7QuY, serial 0 ID "" CA ssh-rsa SHA256:Vcx3pHuh1dhowW4ssS7hXLx7Lqc6sfg9NjmaL8wFo2Y valid after 2024-01-17T10:23:25
debug2: Server host certificate hostname: noah-v15-agentless.root.tele.ottr.sh
debug2: Server host certificate hostname: noah-v15-agentless
debug2: Server host certificate hostname: localhost
debug2: Server host certificate hostname: 127.0.0.1
debug2: Server host certificate hostname: ::1
debug2: Server host certificate hostname: f1a3daa0-3582-48fb-b30a-485cddfe45b2.leaf.tele.ottr.sh
debug2: Server host certificate hostname: 34.171.84.37
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: No matching CA found. Retry with plain key
debug1: hostkeys_find_by_key_hostfile: hostkeys file /etc/ssh/ssh_known_hosts does not exist
debug1: hostkeys_find_by_key_hostfile: hostkeys file /etc/ssh/ssh_known_hosts2 does not exist
The authenticity of host 'noah-v15-agentless.leaf.tele.ottr.sh (<no hostip for proxy command>)' can't be established.
RSA key fingerprint is SHA256:Cwo2MauQbTIO8kH3s3pDAHYUFK8cCXwLvkjCK1r7QuY.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])? 

Bug details:

This was previously discussed as a follow on from https://github.com/gravitational/teleport/issues/31281

strideynet commented 9 months ago

There's two issues here:

// shouldDialAndForward returns whether a connection should be proxied
// and forwarded or not.
func shouldDialAndForward(params reversetunnelclient.DialParams, recConfig types.SessionRecordingConfig) bool {
    // connection is already being tunneled, do not forward
    if params.FromPeerProxy {
        return false
    }
    // the node is an agentless node, the connection must be forwarded
    if params.TargetServer != nil && params.TargetServer.IsOpenSSHNode() {
        return true
    }
    // proxy session recording mode is being used and an SSH session
    // is being requested, the connection must be forwarded
    if params.ConnType == types.NodeTunnel && services.IsRecordAtProxy(recConfig.GetMode()) {
        return true
    }
    return false
}

It looks like this would potentially impact leaf nodes when proxy recording mode is enabled as well.