tsh ssh -A agent forwarding does not forward non-Teleport identities when using proxy recording modes

[Enomised]

Expected behavior:

When a user runs the ssh-add -l command before connecting via Teleport, a list of identities is printed.

When that user runs tsh ssh -A, the -A flag is documented as forwarding the ssh-agent to the target host. So after logging in to the target host, running ssh-add -l should show the same list of identities, via the forwarded agent.

Current behavior:

[localhost] /Users/ddesimone>  ssh-add -l
4096 SHA256:McYSIoixZ2cVwOmdsw2x73mSpeFq+H9tWGUwb1hmAzY ddesimone@FoxBookPro (RSA)
2048 SHA256:ZnYJTE2ve9d2YIKf//xDjTTlP2j24IwLrUEEx+JfmQ4 teleport:david.desimone@example.com (RSA-CERT)
2048 SHA256:ZnYJTE2ve9d2YIKf//xDjTTlP2j24IwLrUEEx+JfmQ4 teleport:david.desimone@example.com (RSA)

The local workstation has three identities. The first is a local identity created by the user, and the next two are certificates established by tsh login

[localhost] /Users/ddesimone>  tsh ssh -A server

[server] /home/ddesimone$  ssh-add -l
2048 SHA256:ZnYJTE2de9d2YIKf//xDjTTlP2j24IKLrUEEx+JfmQ4 teleport:david.desimone@example.com (RSA-CERT)
2048 SHA256:ZnYJTE2de9d2YIKf//xDjTTlP2j24IKLrUEEx+JfmQ4 teleport:david.desimone@example.com (RSA)

Only the teleport certificates are shown. The local identity configured by the user no longer appears. Is this really a connection forwarded to the local user's agent? It's either being filtered, or is a fake agent attempting to resemble the user's agent. Neither case seems like what is documented, a simple forward to the user's agent.

Bug details:

• Teleport version

Teleport v10.3.2 git:v10.3.2-0-g504d64cea go1.18.6
Proxy version: 10.2.2

macOS Monterey 12.6

• Recreation steps

My recreation steps are shown above.

[stevenGravy]

@Enomised can you confirm the role has agent forwarding turned on?

[Enomised]

Is that a server-side setting?

I am not the administrator of the proxy server, so I am not sure how to find that information.

[russjones]

I tried to reproduce and could not, so will close for now.

@Enomised You'll have to have forward_agent: true on at least one of your roles. If that's the case, agent forwarding should work as expected.

See the following example and link for more details.

https://goteleport.com/docs/access-controls/reference/

kind: role
version: v5
metadata:
  name: example
spec:
  options:
    forward_agent: true

[Enomised]

Thanks for looking into this.

Unfortunately, as a low-ranking user, I do not have insight or influence into our server's configuration.

Is this expected behavior, that if forward_agent is false, then tsh ssh -A will cause the agent to appear on the remote side, but only containing the Teleport-related identities?

[russjones]

@Enomised No, you should see all keys loaded at $SSH_AUTH_SOCK.

The Teleport cluster you are running (auth and proxies) are on 10.2.2 right? I tried to reproduce this behavior but could not.

[Enomised]

Yes, my department recently upgraded to 10.2 of Teleport, sorry that I don't have exact version info for you.

I did some more experiments with the agent.

1. Deleted all identities from my local agent with ssh-add -D.
2. tsh ssh -A server to connect to a server. This triggered a login process to create identities, and forwarded them to the server.
3. At this point, both the local client and the server show the same two Teleport identities in response to ssh-add -l.
4. On the server, create a new key with ssh-keygen. Add it to the key agent with ssh-add.
5. On the server, I now see three identities in response to ssh-add -l.
6. But on the client, I still see only the two identities.

It really seems as if the proxy extracts the keys it's interested in forwarding, and creates a new agent instance to forward to my server, and so my client's agent and the server's agent are two distinct agents that do not communicate further.

[sdickhoven]

i am seeing the same issue on a teleport 12.2.4 cluster (tsh client version: 13.0.0) specifically with session_recording: proxy-sync.

if i switch the cluster to node-sync recording mode, everything works as expected.

local agent:

$ ssh-add -l
4096 SHA256:9TNFKZq9ReUjddqurgSFI4VOCRET4nbsXmK3hIAExic <redacted> (RSA)
256 SHA256:ma9vRdqqHbd8HXXbniBzGiGoeXTp9Zcuwr6Mb3UyGdk <redacted> (ED25519)
2048 SHA256:u5tr/SodlORNliRU+o0ouEFhqj0uxB3VNrWNZEseV+k teleport:<redacted>:teleport1-east1-us-dev:<redacted> (RSA-CERT)
2048 SHA256:u5tr/SodlORNliRU+o0ouEFhqj0uxB3VNrWNZEseV+k teleport:<redacted>:teleport1-east1-us-dev:<redacted> (RSA)

proxy-sync:

$ tsh ssh -A server
user@server:~$ ssh-add -l
2048 SHA256:u5tr/SodlORNliRU+o0ouEFhqj0uxB3VNrWNZEseV+k teleport:<redacted>:teleport1-east1-us-dev:<redacted> (RSA-CERT)
2048 SHA256:u5tr/SodlORNliRU+o0ouEFhqj0uxB3VNrWNZEseV+k teleport:<redacted>:teleport1-east1-us-dev:<redacted> (RSA)

node-sync:

$ tsh ssh -A server
user@server:~$ ssh-add -l
4096 SHA256:9TNFKZq9ReUjddqurgSFI4VOCRET4nbsXmK3hIAExic <redacted> (RSA)
256 SHA256:ma9vRdqqHbd8HXXbniBzGiGoeXTp9Zcuwr6Mb3UyGdk <redacted> (ED25519)
2048 SHA256:u5tr/SodlORNliRU+o0ouEFhqj0uxB3VNrWNZEseV+k teleport:<redacted>:teleport1-east1-us-dev:<redacted> (RSA-CERT)
2048 SHA256:u5tr/SodlORNliRU+o0ouEFhqj0uxB3VNrWNZEseV+k teleport:<redacted>:teleport1-east1-us-dev:<redacted> (RSA)

my role permits agent forwarding (otherwise proxy-sync recording mode wouldn't work).

so i'm guessing that the agent forwarding on the proxy is doing some funny business.

[webvictim]

As noted, this is not an issue (at least on Teleport 14.3.0) when using node or node-sync recording modes - all keys in the running agent will be forwarded correctly using tsh ssh -A user@host.

Changing the title of the issue accordingly.

[aprohorov-callsign]

No updates yet? Faced with the same issue