search

gravitational / teleport

The easiest, and most secure way to access and protect all of your infrastructure.
https://goteleport.com
GNU Affero General Public License v3.0
17.69k stars 1.77k forks source link

`require_session_mfa: hardware_key_touch` buggy `tsh login` behavior #20384

Open GavinFrazar opened 1 year ago

GavinFrazar commented 1 year ago

Expected behavior: $ tsh login logs in with mfa and yubikey prompts.

Current behavior:

$ tsh login --user=martian
Enter password for Teleport user martian:
Tap any security key
Tap your YubiKey
ERROR: ssh: handshake failed: command failed: transmitting request: an attempt was made to end a non-existent transaction

Bug details:

Steps to reproduce:

There’s a brief window of time after you tap for the yubikey prompt (too early, before it starts blinking) where the key will be blinking, but tapping during this window is futile and you still get ssh: handshak failed:… error message.

It seems that to successfully login, you must be careful to wait for the yubikey to start blinking again before tapping it the second time. We should at least hint the user about the quirky behavior when they see the error message.

Additionally, after you have reproduced this failure, the yubikey will still be blinking (for a few seconds). If you try to log in again while it's still blinking, you get another error. We should try to provide a hint in this case as well to simply wait for the key to stop blinking and try again. I occasionally ran into an issue where I would start seeing this error message as well:

ERROR: failed to get a YubiKey private key
connecting to smart card: the operation requires a Smart Card, but no Smart Card is currently in the device

however I cannot reproduce this one. It only resolved when I unplugged my yubikey and plugged it back in.

Debug logs

initial login attempt:

$ tsh login --user=martian -d
[CLIENT]    INFO no host login given. defaulting to gavin client/api.go:935
[CLIENT]    INFO [KEY AGENT] Connected to the system agent: "/private/tmp/com.apple.launchd.YiiInsIMPF/Listeners" client/api.go:3880
[CLIENT]    DEBU attempting to use loopback pool for local proxy addr: mars.local.gd:443 client/api.go:3838
[CLIENT]    DEBU reading self-signed certs from: /var/lib/teleport/webproxy_cert.pem client/api.go:3846
[CLIENT]    DEBU could not open any path in: /var/lib/teleport/webproxy_cert.pem client/api.go:3850
            DEBU Attempting GET mars.local.gd:443/webapi/ping webclient/webclient.go:129
[CLIENT]    DEBU Attempting to login with YubiKey private key. client/api.go:3256
Enter password for Teleport user martian:
[CLIENT]    DEBU attempting to use loopback pool for local proxy addr: mars.local.gd:443 client/api.go:3838
[CLIENT]    DEBU reading self-signed certs from: /var/lib/teleport/webproxy_cert.pem client/api.go:3846
[CLIENT]    DEBU could not open any path in: /var/lib/teleport/webproxy_cert.pem client/api.go:3850
[CLIENT]    DEBU HTTPS client init(proxyAddr=mars.local.gd:443, insecure=false, extraHeaders=map[]) client/weblogin.go:259
[CLIENT]    DEBU WebAuthn: prompting devices with origin "https://mars.local.gd:443" client/mfa.go:225
            DEBU Attempting platform login webauthncli/api.go:141
            DEBU Platform login failed, falling back to cross-platform error:[touch ID not available] webauthncli/api.go:147
            DEBU FIDO2: Using libfido2 for assertion webauthncli/api.go:157
            DEBU FIDO2: assertion: passwordless=false, uv=false webauthncli/fido2.go:119
Tap any security key
            DEBU FIDO2: Info for device ioreg://4295300941: &libfido2.DeviceInfo{Versions:[]string{"U2F_V2", "FIDO_2_0", "FIDO_2_1_PRE"}, Extensions:[]string{"credProtect", "hmac-secret"}, AAGUID:[]uint8{0x2f, 0xc0, 0x57, 0x9f, 0x81, 0x13, 0x47, 0xea, 0xb1, 0x16, 0xbb, 0x5a, 0x8d, 0xb9, 0x20, 0x2a}, Options:[]libfido2.Option{libfido2.Option{Name:"rk", Value:"true"}, libfido2.Option{Name:"up", Value:"true"}, libfido2.Option{Name:"plat", Value:"false"}, libfido2.Option{Name:"clientPin", Value:"false"}, libfido2.Option{Name:"credentialMgmtPreview", Value:"true"}}, Protocols:[]uint8{0x2, 0x1}} webauthncli/fido2.go:879
            DEBU FIDO2: Found 1 new devices webauthncli/fido2.go:888
            DEBU FIDO2: Got 1 assertions webauthncli/fido2.go:200
            DEBU FIDO2: Authenticated: credential ID (b64) = kx80tch_hvAdZL2boic77HiUUUuoGjmIIn0varEA1yCIc8EJog2NiARdwYc1YDrlT16Cl6beqT7nYRo0Ylp5NQ, user ID (hex) = , user name = "" webauthncli/fido2.go:208
            DEBU FIDO2: device ioreg://4295300941: selected with err=<nil> webauthncli/fido2.go:908
[KEYSTORE]  DEBU Adding known host mars.local.gd with proxy mars.local.gd client/trusted_certs_store.go:385
[KEYSTORE]  DEBU Adding known host mars.local.gd with proxy mars.local.gd client/trusted_certs_store.go:385
[KEYAGENT]  INFO Loading SSH key for user "martian" and cluster "mars.local.gd". client/keyagent.go:195
[KEYAGENT]  INFO Skipping adding key to SSH system agent for non-standard key type *keys.YubiKeyPrivateKey client/keyagent.go:201
[KEYSTORE]  DEBU Teleport TLS certificate valid until "2023-01-19 14:18:06 +0000 UTC". client/client_store.go:89
[KEYSTORE]  DEBU Teleport TLS certificate valid until "2023-01-19 14:18:06 +0000 UTC". client/client_store.go:89
[CLIENT]    INFO Connecting to proxy=mars.local.gd:443 login="gavin" using TLS Routing client/api.go:2775
[KEYSTORE]  DEBU Teleport TLS certificate valid until "2023-01-19 14:18:06 +0000 UTC". client/client_store.go:89
[HTTP:PROX] DEBU No proxy set in environment, returning direct dialer. proxy/proxy.go:301
[KEYSTORE]  DEBU Teleport TLS certificate valid until "2023-01-19 14:18:06 +0000 UTC". client/client_store.go:89
[KEYAGENT]  DEBU "Checking key: ssh-rsa-cert-v01@openssh.com AAAAHHNzaC1yc2EtY2VydC12MDFAb3BlbnNzaC5jb20AAAAg1KTxYSIuHeUSo8v/s/o3ZPo0bPHirxfcF08cP7Bgp5IAAAADAQABAAABAQC6MJic5VxJgxqnuBqaQQTZgV/hzM7i+3awvjqlMmgZuVoD/aL6VszG+XdjNe8nYJ4YB1bLCkIKykKc4kn4SdPAITgHf+cJ4zz0gn28c428e9mRCneb2JNdPSIOQjrZiCOWb/vEOKU/AKhhvvaNkbWIHiUOkWe39X0vclahNOrM+LMAnSlwI9/UKWrgayNcfVVgBwt6uCfMtyvtIi7vQe1mVCBS1yng0wBn3Qrgug6Z/3G/5xrxCH6CL2LHiNCeS7dowtZ9EEE05PU774o0Vx2tjGFWuK2P+/aBUyQSOyyGspVLA7SMHdE/VzR0R8n7VQvkgYPOMItC3wynMBa0UZpxAAAAAAAAAAAAAAACAAAAAAAAANsAAAAyYzIzN2YxZTgtMzU4Zi00NTA3LTk0ZjEtMWZhMGNmZGJmNDkzLm1hcnMubG9jYWwuZ2QAAAAkYzIzN2YxZTgtMzU4Zi00NTA3LTk0ZjEtMWZhMGNmZGJmNDkzAAAAG21hcnMubG9jYWwuZ2QubWFycy5sb2NhbC5nZAAAAA1tYXJzLmxvY2FsLmdkAAAACWxvY2FsaG9zdAAAAAkxMjcuMC4wLjEAAAADOjoxAAAAKHJlbW90ZS5rdWJlLnByb3h5LnRlbGVwb3J0LmNsdXN0ZXIubG9jYWwAAAAAY8iUzf//////////AAAAAAAAAE0AAAAUeC10ZWxlcG9ydC1hdXRob3JpdHkAAAARAAAADW1hcnMubG9jYWwuZ2QAAAAPeC10ZWxlcG9ydC1yb2xlAAAACQAAAAVQcm94eQAAAAAAAAEXAAAAB3NzaC1yc2EAAAADAQABAAABAQDCpG4T/kH6uGA6kamtMxBACNrPYvGccTIhLn2eMcrFlm0dTNQf6M4mBK4rt1DlmwcCICK0yVL8faVbWSXG7FfOVMm9BwIedDrCB5euloc+2SH2YCw+/37rz10cYK3vv43dD4e84xsFdM2fnYePVFiqC77feZrbX1rOiVsgllzZBE5YQhACje/zOZvm6rNs6rqq65RFkk4xCIVQpkFRaVgv0jPGirI49pStL7z2aAnP8i+zSiqaPVUyM72JyEa3KjRPHYTO50gdHHxuO+uHyk9HFDHNmYIJJKN1QdUM7h9FgipP9tNAzoGuCUueWf5a3CWOYagxx6iRfAtAk4STYR33AAABFAAAAAxyc2Etc2hhMi01MTIAAAEAd/7yfBxIsdA0/kCz2BRhVyKQhjbKZbmDigg/Sf/NmYHdfxh9ieHrWCxK/fNwV7bqQACu+rc5UJhE5loD9myxMNS6A3DrBj7P+J35aP9GRHHeYTXKP7Uig1NERd0/Vh3g7+6BzaJnHk44RG1TelUYQhHuxyUON2MGW9v/RrdYOT8u6z8FfpL0U5KWy/F4HizAAtBu5G9fAIpmR2MN+W2/mwAinVPUhZFVoX48ABoE4bln18oFVeT8NwqoSoxDeXsDIZWvwn7Y5cusEwa0gwJzLXxrxwYRMADulrcoWDMGc2+wMwda/ufZ4lMifdgs6ECXs38hYirk4BYCop3F5eiSdw==\n." client/keyagent.go:367
[KEYAGENT]  DEBU Validated host mars.local.gd:443. client/keyagent.go:373
Tap your YubiKey

ERROR REPORT:
Original Error: *errors.errorString ssh: handshake failed: command failed: transmitting request: an attempt was made to end a non-existent transaction
Stack Trace:
    github.com/gravitational/teleport/api@v0.0.0/observability/tracing/ssh/ssh.go:148 github.com/gravitational/teleport/api/observability/tracing/ssh.NewClientConn
    github.com/gravitational/teleport/api@v0.0.0/observability/tracing/ssh/ssh.go:161 github.com/gravitational/teleport/api/observability/tracing/ssh.NewClientConnWithDeadline
    github.com/gravitational/teleport/lib/utils/proxy/proxy.go:85 github.com/gravitational/teleport/lib/utils/proxy.directDial.dialALPNWithDeadline
    github.com/gravitational/teleport/lib/utils/proxy/proxy.go:120 github.com/gravitational/teleport/lib/utils/proxy.directDial.Dial
    github.com/gravitational/teleport/lib/client/api.go:2834 github.com/gravitational/teleport/lib/client.makeProxySSHClientWithTLSWrapper
    github.com/gravitational/teleport/lib/client/api.go:2776 github.com/gravitational/teleport/lib/client.makeProxySSHClient
    github.com/gravitational/teleport/lib/client/api.go:2730 github.com/gravitational/teleport/lib/client.(*TeleportClient).connectToProxy
    github.com/gravitational/teleport/lib/client/api.go:2655 github.com/gravitational/teleport/lib/client.(*TeleportClient).ConnectToProxy.func1
    runtime/asm_amd64.s:1594 runtime.goexit
User Message: ssh: handshake failed: command failed: transmitting request: an attempt was made to end a non-existent transaction

Subsequent attempt while yubikey is still blinking:

$ tsh login --user=martian -d
INFO [CLIENT]    no host login given. defaulting to gavin client/api.go:935
INFO [CLIENT]    [KEY AGENT] Connected to the system agent: "/private/tmp/com.apple.launchd.YiiInsIMPF/Listeners" client/api.go:3880
DEBU [CLIENT]    attempting to use loopback pool for local proxy addr: mars.local.gd:443 client/api.go:3838
DEBU [CLIENT]    reading self-signed certs from: /var/lib/teleport/webproxy_cert.pem client/api.go:3846
DEBU [CLIENT]    could not open any path in: /var/lib/teleport/webproxy_cert.pem client/api.go:3850
DEBU             Attempting GET mars.local.gd:443/webapi/ping webclient/webclient.go:129

ERROR REPORT:
Original Error: *retryutils.permanentRetryError selecting piv applet: command failed: transmitting request: an attempt was made to end a non-existent transaction
Stack Trace:
    github.com/gravitational/teleport/api@v0.0.0/utils/retryutils/retry.go:189 github.com/gravitational/teleport/api/utils/retryutils.(*Linear).For
    github.com/gravitational/teleport/api@v0.0.0/utils/keys/yubikey.go:352 github.com/gravitational/teleport/api/utils/keys.(*yubiKey).open
    github.com/gravitational/teleport/api@v0.0.0/utils/keys/yubikey.go:252 github.com/gravitational/teleport/api/utils/keys.newYubiKey
    github.com/gravitational/teleport/api@v0.0.0/utils/keys/yubikey.go:396 github.com/gravitational/teleport/api/utils/keys.findYubiKey
    github.com/gravitational/teleport/api@v0.0.0/utils/keys/yubikey.go:127 github.com/gravitational/teleport/api/utils/keys.parseYubiKeyPrivateKeyData
    github.com/gravitational/teleport/api@v0.0.0/utils/keys/privatekey.go:205 github.com/gravitational/teleport/api/utils/keys.ParsePrivateKey
    github.com/gravitational/teleport/api@v0.0.0/utils/keys/privatekey.go:236 github.com/gravitational/teleport/api/utils/keys.ParseKeyPair
    github.com/gravitational/teleport/api@v0.0.0/utils/keys/privatekey.go:227 github.com/gravitational/teleport/api/utils/keys.LoadKeyPair
    github.com/gravitational/teleport/lib/client/keystore.go:310 github.com/gravitational/teleport/lib/client.(*FSKeyStore).GetKey
    github.com/gravitational/teleport/lib/client/client_store.go:80 github.com/gravitational/teleport/lib/client.(*Store).GetKey
    github.com/gravitational/teleport/lib/client/keyagent.go:298 github.com/gravitational/teleport/lib/client.(*LocalKeyAgent).GetKey
    github.com/gravitational/teleport/lib/client/keyagent.go:313 github.com/gravitational/teleport/lib/client.(*LocalKeyAgent).GetCoreKey
    github.com/gravitational/teleport/lib/client/api.go:3241 github.com/gravitational/teleport/lib/client.(*TeleportClient).GetNewLoginKey
    github.com/gravitational/teleport/lib/client/api.go:3181 github.com/gravitational/teleport/lib/client.(*TeleportClient).SSHLogin
    github.com/gravitational/teleport/lib/client/api.go:3024 github.com/gravitational/teleport/lib/client.(*TeleportClient).Login
    github.com/gravitational/teleport/tool/tsh/tsh.go:1578 main.onLogin
    github.com/gravitational/teleport/tool/tsh/tsh.go:1071 main.Run
    github.com/gravitational/teleport/tool/tsh/tsh.go:472 main.main
    runtime/proc.go:250 runtime.main
    runtime/asm_amd64.s:1594 runtime.goexit
User Message: failed to parse YubiKey private key
    selecting piv applet: command failed: transmitting request: an attempt was made to end a non-existent transaction
codingllama commented 1 year ago

Assigning @Joerger for PIV support.

I think the endgame here is to do only the PIV check, skipping MFA entirely. Yubis, in my experience, don't do well with operations that close in time. Other than that, which may be a more difficult change, I'm not sure there's much that can be done.

Joerger commented 1 year ago

FWIW I was not able to reproduce this with my YubiKey 5C NFC or my YubiKey 5 Nano on Linux.

codingllama commented 1 year ago

Hey @GavinFrazar and @Joerger, did we get a resolution here?

Joerger commented 1 year ago

Hey @GavinFrazar and @Joerger, did we get a resolution here?

No, I'm not sure we have a good solution as of now. We don't have eyes into the transactions on the YubiKey, so like you said, it can be finicky when multiple transactions are being made, especially from different protocols.

IIUC the workaround is to just tap the YubiKey a bit slower. We could try to build a timeout into tsh to prevent it from prompting them one after another too quickly, but I'm not sure an arbitrary timeout is guaranteed to help here.

codingllama commented 1 year ago

I still think "combining" the prompts into just the PIV prompt would be the best call. If there are session "keepalive" prompts, further from the initial one, then only those should do WebAuthn.

Let's keep the issue around, then. Thanks for the update.

Joerger commented 1 year ago

I've added a troubleshooting tip for this error for now.