search

gravitational / teleport

The easiest, and most secure way to access and protect all of your infrastructure.
https://goteleport.com
GNU Affero General Public License v3.0
17.5k stars 1.75k forks source link

gpg-agent does not support SSH certificates, tsh should be aware of this #3169

Closed webvictim closed 3 years ago

webvictim commented 4 years ago

What happened:

What you expected to happen: 1)tsh should have a command-line flag which disables any attempt to write to a running agent 2) Ideally, tsh would be able to detect when gpg-agent is running rather than ssh-agent and deliberately avoid writing its keys there.

This would enable much smoother Teleport operations for Yubikey users. tsh still writes its certificates to ~/.tsh/keys/<cluster>, which means that people can use these for their Teleport operations instead.

How to reproduce it (as minimally and precisely as possible): Run gpg-agent, log into a Teleport cluster using tsh then observe that what's in the agent is not a valid certificate (error fetching identities: Invalid key length)

Environment:

russjones commented 4 years ago

In a recent discussion with a customer: 

Sure. gpg-agent works with our YubiKeys so when we have individual users SSH keys,
they would be stored on a yubikey. gpg-agent is the agent that connects between the
YubiKey and acts as an ssh agent. The normal ssh-agent doesn't have such a capability.
So, we would switch ssh-agent out for gpg-agent. When we do that, when tsh adds the
ssh certificate to gpg-agent, it doesn't know how to handle it (It adds the cert, but
then will no longer serve up any identities, effectively killing the gpg-agent)
thardie commented 4 years ago

Interestingly, on the YubiKey side, at least 1 other user claims there is a way to make this work: https://github.com/drduh/YubiKey-Guide/issues/43

benarent commented 4 years ago

This came up again with Reliability Improvements for E- Adding other customer to the issue.

aelkugia commented 4 years ago

Had another conversation with a customer interested in this, who has some engineers using gpg-agent/yubikey.

cpanato commented 4 years ago

+1

cnelson commented 4 years ago

Also ran into this. currently working around it with alias tsh='SSH_AUTH_SOCK= tsh'

awly commented 4 years ago

Fix is in review.

FWIW, it appears that OpenSSH 8.2+ supports security keys now: https://buttondown.email/cryptography-dispatches/archive/cryptography-dispatches-openssh-82-just-works/. I haven't tried it myself, but might be interesting for folks in here.

eddiewang commented 4 years ago

Tried using the --use-local-ssh-agent flag with no avail on Teleport v4.4.0-dev git:v4.2.0-alpha.5-478-gb1280007e go1.14.2

Still getting error fetching identities: Invalid key length after a successful login. Using gpg-agent with Yubikey.

webvictim commented 4 years ago

@eddiewang You want --no-use-local-ssh-agent

eddiewang commented 4 years ago

@webvictim lol thanks. that deserves a :man_facepalming:

webvictim commented 4 years ago

No worries - it's a bit of a confusing flag. We've written some documentation explaining it, but that won't be coming out until Teleport 4.3 does (which will be very soon)

klizhentas commented 3 years ago

Let's give it another go in 5.1

  1. Can we detect gpg-agent and not load the certs?
  2. Can we reduce the amount of flags? There are other requests, for example load agent, so there are 4 states possible. Can we use one flag with values or may be there is another, more creative way. The less flags the better.
klizhentas commented 3 years ago

Can we patch gpg-agent upstream to support certs?

Chili-Man commented 3 years ago

Is there anyway to disable tsh login from generating an SSH key when we only use teleport for kubernetes access?

And for those wondering how to resolve the error by removing the offending keys; they can be manually removed from within the ~/.gnupg/private-keys-v1.d directory

webvictim commented 3 years ago

There's not currently a way to stop SSH certs from being generated completely, but you can prevent tsh from attempting to add them to the agent by using tsh login --no-use-local-ssh-agent or setting the TELEPORT_USE_LOCAL_SSH_AGENT environment variable to false.

Chili-Man commented 3 years ago

Even using tsh login --no-use-local-ssh-agent doesn't seem to help; after I login, and then I try to use the loaded kubeconfig provided through that command, by invoking any kubectl command, its added back into the ssh agent when kubectl tryies to get the credentials with the command in kubeconfig tsh kube credentials --kube-cluster=teleport.example.com --teleport-cluster=teleport.example.com

Chili-Man commented 3 years ago

After further testing, it appears that if you manually add the --no-use-local-ssh-agent as part of the command it uses in the kubeconfig to get the credentials, it won't add the ssh key to your gpg-agent. Is there a way to have that flag automatically added?

it appears that you can actually export TELEPORT_USE_LOCAL_SSH_AGENT=false in your shell's profile or rc file, where it will preserve the effect of not adding the tsh ssh keys to your gpg-agent

webvictim commented 3 years ago

@Chili-Man What happens if you run export TELEPORT_USE_LOCAL_SSH_AGENT=false (or whatever is appropriate for your shell) and then try? Does the embedded tsh command honour that? (it should)

Chili-Man commented 3 years ago

@webvictim thanks, that works! I've updated my comment above to reflect that.

klizhentas commented 3 years ago

@awly please look into the issue when working on U2F support for tsh. In addition to that we have requests for people who don't want the keys to be written to disk and stored only in memory in agent.

I think we have to build a matrix of all possible combinations we are going to support and provide some UX that integrates well with U2f support you are adding.

klizhentas commented 3 years ago

https://github.com/gravitational/teleport/issues/4863

natikgadzhi commented 3 years ago

+1. Just shot myself in the foot with this, too.

thardie commented 3 years ago

The only reason I was using gpg-agent was to use a yubikey GPG based auth for SSH. You can instead use PIV with Yubikey to achieve similar functionality, then use the regular SSH Agent instead of gpg-agent. See here for details on how to set this up: https://developers.yubico.com/PIV/Guides/SSH_with_PIV_and_PKCS11.html

Using this method then lets teleport and yubikey coexist properly with a regular SSH Agent.

Chili-Man commented 3 years ago

@thardie unfortunately, it appears that PIV only works if your gpg key was an RSA 2048-bit key 😞

https://github.com/Yubico/yubico-piv-tool/issues/58

Seems to be an underlying issue with the specification for PIV

webvictim commented 3 years ago

Related to #5487

santiago-mooser commented 1 month ago

Sorry to comment on a closed issue, but this is happening again with Teleport Connect. However, this only happens when carrying out SSH operations outside of Teleport Connect. Teleport Conect itself works fine, as does tsh.

e.g. We'r using yubikeys that use gpg-agent to authenticate to git, so when running a git fetch usually we'd be able to authenticate with a yubikey and pulling the updates from the repo. However, if we launch Teleport Connect, it'll add new keys to the ~/.gnupg/private-keys-v1.d/ folder and cause issues with the git operation:

debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: get_agent_identities: ssh_fetch_identitylist: Invalid key length
debug1: Will attempt key: /Users/username/.ssh/id_rsa
debug1: Will attempt key: /Users/username/.ssh/id_ecdsa
debug1: Will attempt key: /Users/username/.ssh/id_ecdsa_sk
debug1: Will attempt key: /Users/username/.ssh/id_ed25519
debug1: Will attempt key: /Users/username/.ssh/id_ed25519_sk
debug1: Will attempt key: /Users/username/.ssh/id_xmss
debug1: Will attempt key: /Users/username/.ssh/id_dsa
debug1: Trying private key: /Users/username/.ssh/id_rsa
debug1: Trying private key: /Users/username/.ssh/id_ecdsa
debug1: Trying private key: /Users/username/.ssh/id_ecdsa_sk
debug1: Trying private key: /Users/username/.ssh/id_ed25519
debug1: Trying private key: /Users/username/.ssh/id_ed25519_sk
debug1: Trying private key: /Users/username/.ssh/id_xmss
debug1: Trying private key: /Users/username/.ssh/id_dsa
debug1: No more authentication methods to try.
git@domain.tld: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

I don't know whether this has to do with the key length of the server, but openssh refuses to work with keys of length <1024: https://www.openssh.com/txt/release-7.6

The keys added by teleport connect to ~/.gnupg/private-keys-v1.d/ seem to have a length of only 1024 bits, which might be related to the issue:

wc ~/.gnupg/private-keys-v1.d/*
      32      40    2262 /Users/username/.gnupg/private-keys-v1.d/02C7149F9848E8E8B31FF45BF606B09FE055D72C.key
      32      40    2266 /Users/username/.gnupg/private-keys-v1.d/14DC5FA73FE0A29CF0254CB0D07874E732DE9979.key
      17      24    1174 /Users/username/.gnupg/private-keys-v1.d/239393F0CFF1FF4693C18238330EDF046AC1D1FC.key
      17      24    1174 /Users/username/.gnupg/private-keys-v1.d/2BBD2A9775E18D657DF3CBD31641EF2D2D60115D.key
      17      24    1174 /Users/username/.gnupg/private-keys-v1.d/4E6C9CE723CD0DD11E16077EA6C35E990CAFD76A.key
      17      24    1174 /Users/username/.gnupg/private-keys-v1.d/90589F05018180CA70A80C8E3AEC0ABB513B0CDC.key
      17      24    1174 /Users/username/.gnupg/private-keys-v1.d/A6D1F7B1B5A1AE187C0BDA12D04E8C14E1E91F47.key
      32      40    2266 /Users/username/.gnupg/private-keys-v1.d/C42AC1C46F1D4E211C735CC7DFAD4FF8391110E9.key
      17      24    1174 /Users/username/.gnupg/private-keys-v1.d/F28808D6BC7B8C8E484EB4F4F41DAB0D3886BECA.key

tsh works just fine

Version numbers

OS: MacOS Sonoma 14.7 Teleport Connect v15.2.5 SSH OpenSSH_9.7p1, LibreSSL 3.3.6 gpg: (GnuPG/MacGPG2) 2.2.32 libgcrypt 1.8.8

ravicious commented 1 month ago

@santiago-mooser Thanks for a comprehensive report. Could you check if connecting to a node through tsh ssh with the --forward-agent flag also breaks your setup?

We think it's because Teleport Connect adds that flag to all SSH connections made from Connect. We're thinking of making it a configurable option in v16.4.x and v15.4.x and then turning it into an opt-in rather than an opt-out setting in v17.

santiago-mooser commented 1 month ago

We don't currently have any SSH servers set up (only web applications, EKS and DBs), but we're adding the servers this month so I'll let you know as soon as onboard them to teleport

It would be great if we could toggle it from the UI! This would fix the issue for all of our devs.

ravicious commented 1 month ago

Thanks for clarifying, it seems that I've made a mistake. Both you and another user on Zendesk who made a similar report did not really need to SSH into a node to trigger this problem. So I'm not sure how much this is related to SSH agent forwarding.

Though there is yet another issue, https://github.com/gravitational/teleport/issues/11662, which I think describes the same problem that you're having where accessing a node with agent forwarding is one of the repro steps.

Alas, I suspect it might be due to tsh (and thus Connect) incorrectly guessing that your agent supports SSH certs, basically replicating the original problem described in this issue.

When you run into the problem with git fetch, what is your $SSH_AUTH_SOCK at that point? tsh checks if it contains gpg-agent when deciding whether to add keys to the agent or not.

santiago-mooser commented 2 weeks ago

Yes, it seems like tsh is guessing that the agent supports SSH certs. I think there's also a restriction on MacOS specifically about the key length (it only supports >=2048 bit keys nad I think the SSH certs added by teleport are 1028 bits?).

I also asked the developer what his SSH_AUTH_SOCK was and he replied that this was what whs being used: /private/tmp/com.apple.launchd.2ErIMl6caK/Listeners

I've since asked the developer to add export SSH_AUTH_SOCK=$(gpgconf --list-dirs agent-ssh-socket) to their rc file so let's see if this fixes the issue since that should change the variable to be a more standard /Users/username/.gnupg/S.gpg-agent.ssh

Any chance there's any updates on adding the --no-use-local-ssh-agent to teleport connect?

ravicious commented 2 weeks ago

I also asked the developer what his SSH_AUTH_SOCK was and he replied that this was what whs being used: /private/tmp/com.apple.launchd.2ErIMl6caK/Listeners

I've since asked the developer to add export SSH_AUTH_SOCK=$(gpgconf --list-dirs agent-ssh-socket) to their rc file so let's see if this fixes the issue since that should change the variable to be a more standard /Users/username/.gnupg/S.gpg-agent.ssh

Thanks, please do let us know if this had any effect, it should help with fixing the issue here.

Any chance there's any updates on adding the --no-use-local-ssh-agent to teleport connect?

I just pushed a PR that adds sshAgent.addKeysToAgent to Connect (https://github.com/gravitational/teleport/pull/47270) which is the equivalent of the --add-keys-to-agent flag from tsh which superseded --use-local-ssh-agent and --no-use-local-ssh-agent. Once it's released, setting this option to "no" should serve as a workaround for this issue.

That being said, I still don't understand how exactly tsh manages to break gpg-agent. If SSH_AUTH_SOCK doesn't point to gpg-agent, then I assume this means that gpg-agent is used only by git. If gpg-agent is used only by git, then I don't know how tsh still manages to write stuff to ~/.gnupg/private-keys-v1.d/.

Our logic of looking at SSH_AUTH_SOCK to check if it contains gpg-agent seems mostly correct. This assumes that the user set up gpg-agent globally by doing some equivalent of gpg-agent --daemon --enable-ssh-support --sh followed by setting the env variables returned by this command. On macOS, there are several caveats regarding desktop apps since those do not read from shell files.