search

gravitational / teleport

The easiest, and most secure way to access and protect all of your infrastructure.
https://goteleport.com
GNU Affero General Public License v3.0
17.5k stars 1.75k forks source link

Machine ID: IAM token attempts to use TLS dialing on non-multiplex config #20058

Closed programmerq closed 9 months ago

programmerq commented 1 year ago

Expected behavior:

IAM join method should work when TLS multiplexing is disabled.

Current behavior:

If a cluster has tls routing disabled, and tbot is trying to use the iam-join methed, it will still attempt to dial using ALPN teleport-proxy-grpc,h2. This causes an ALB to return HTTP status code of "464", which indicates that an unsupported protocol was requested.

I also found that bypassing an ALB would also cause a failure, using the following config file:

teleport.yaml for lab cluster This was used on my laptop with `teleport.example.com` pointed to 127.0.0.1 for this test, along with a cert that I imported to my OS trust store. ```yaml version: v3 teleport: nodename: cluster log: output: stderr severity: INFO format: output: text ca_pin: "" diag_addr: "" auth_service: enabled: "yes" listen_addr: 0.0.0.0:3025 cluster_name: example license_file: /var/lib/teleport/license.pem proxy_listener_mode: separate ssh_service: enabled: "yes" labels: env: example commands: - name: hostname command: [hostname] period: 1m0s proxy_service: enabled: "yes" web_listen_addr: 0.0.0.0:3080 public_addr: teleport.example.com:3080 https_keypairs: - key_file: /var/lib/teleport/cert-key.pem cert_file: /var/lib/teleport/cert.pem acme: {} listen_addr: 0.0.0.0:3023 tunnel_listen_addr: 0.0.0.0:3024 ``` Here's the output of `curl https://teleport.example.com:3080/webapi/ping`: ```json { "auth": { "type": "local", "second_factor": "otp", "preferred_local_mfa": "otp", "local": { "name": "" }, "private_key_policy": "none", "has_motd": false }, "proxy": { "kube": { "enabled": true }, "ssh": { "listen_addr": "[::]:3023", "tunnel_listen_addr": "0.0.0.0:3024", "web_listen_addr": "0.0.0.0:3080", "public_addr": "teleport.example.com:3080" }, "db": {}, "tls_routing_enabled": false }, "server_version": "11.1.4", "min_client_version": "10.0.0", "cluster_name": "00015067" } ```

Following the machine id IAM join instructions from the docs, I created an iam-token and a bot. This appears to complete as expected.

tbot setup info ```yaml kind: token version: v2 metadata: name: iam-token expires: "2024-01-01T00:00:00Z" spec: roles: [Bot] join_method: iam bot_name: iambot allow: - aws_account: "111111111111" ``` ``` $ tctl create -f token.yaml $ tctl bots add iambot --token iam-token --roles access --logins=iambot ```
tbot logs including stacktrace ``` $ aws sts get-caller-identity # this works $ tbot start --data-dir=/var/lib/teleport/bot --destination-dir=/var/lib/teleport/dest --token=iam-token --join-method=iam --auth-server=teleport.example.com:3080 WARN [TBOT] CLI parameters are overriding onboarding config from config/config.go:378 WARN [TBOT] Secure symlinks not supported on this platform, set `symlinks: insecure` to disable this message/var/lib/teleport/bot/key-cert.pub botfs/fs_other.go:47 WARN [TBOT] Secure symlinks not supported on this platform, set `symlinks: insecure` to disable this message/var/lib/teleport/bot/sshcert botfs/fs_other.go:47 WARN [TBOT] Secure symlinks not supported on this platform, set `symlinks: insecure` to disable this message/var/lib/teleport/bot/.write-test botfs/fs_other.go:71 INFO [TBOT] Attempting to generate new identity from token tbot/renew.go:455 INFO [AUTH] Attempting registration via proxy server. auth/register.go:245 INFO [AUTH] Attempting to register Bot with IAM method using regional STS endpoint auth/register.go:565 INFO [AUTH] Failed to register Bot using regional STS endpoint error:[ ERROR REPORT: Original Error: *status.Error rpc error: code = Unknown desc = rpc error: code = InvalidArgument desc = missing parameter HostID Stack Trace: github.com/gravitational/teleport/api@v0.0.0/client/joinservice.go:83 github.com/gravitational/teleport/api/client.(*JoinServiceClient).RegisterUsingIAMMethod github.com/gravitational/teleport/lib/auth/register.go:567 github.com/gravitational/teleport/lib/auth.registerUsingIAMMethod github.com/gravitational/teleport/lib/auth/register.go:287 github.com/gravitational/teleport/lib/auth.registerThroughProxy github.com/gravitational/teleport/lib/auth/register.go:246 github.com/gravitational/teleport/lib/auth.Register github.com/gravitational/teleport/lib/tbot/renew.go:477 github.com/gravitational/teleport/lib/tbot.(*Bot).getIdentityFromToken github.com/gravitational/teleport/lib/tbot/tbot.go:357 github.com/gravitational/teleport/lib/tbot.(*Bot).initialize github.com/gravitational/teleport/lib/tbot/tbot.go:228 github.com/gravitational/teleport/lib/tbot.(*Bot).Run github.com/gravitational/teleport/tool/tbot/main.go:234 main.onStart github.com/gravitational/teleport/tool/tbot/main.go:153 main.Run github.com/gravitational/teleport/tool/tbot/main.go:48 main.main runtime/proc.go:250 runtime.main runtime/asm_amd64.s:1594 runtime.goexit User Message: rpc error: code = Unknown desc = rpc error: code = InvalidArgument desc = missing parameter HostID] auth/register.go:590 INFO [AUTH] Attempting to register Bot with IAM method using global STS endpoint auth/register.go:565 INFO [AUTH] Attempting to use the global STS endpoint for the IAM join method. This will probably fail in non-default AWS partitions such as China or GovCloud, or if FIPS mode is enabled. Consider setting the AWS_REGION environment variable, setting the region in ~/.aws/config, or enabling the IMDSv2. auth/join_iam.go:503 INFO [AUTH] Failed to register Bot using global STS endpoint error:[ ERROR REPORT: Original Error: *status.Error rpc error: code = Unknown desc = rpc error: code = InvalidArgument desc = missing parameter HostID Stack Trace: github.com/gravitational/teleport/api@v0.0.0/client/joinservice.go:83 github.com/gravitational/teleport/api/client.(*JoinServiceClient).RegisterUsingIAMMethod github.com/gravitational/teleport/lib/auth/register.go:567 github.com/gravitational/teleport/lib/auth.registerUsingIAMMethod github.com/gravitational/teleport/lib/auth/register.go:287 github.com/gravitational/teleport/lib/auth.registerThroughProxy github.com/gravitational/teleport/lib/auth/register.go:246 github.com/gravitational/teleport/lib/auth.Register github.com/gravitational/teleport/lib/tbot/renew.go:477 github.com/gravitational/teleport/lib/tbot.(*Bot).getIdentityFromToken github.com/gravitational/teleport/lib/tbot/tbot.go:357 github.com/gravitational/teleport/lib/tbot.(*Bot).initialize github.com/gravitational/teleport/lib/tbot/tbot.go:228 github.com/gravitational/teleport/lib/tbot.(*Bot).Run github.com/gravitational/teleport/tool/tbot/main.go:234 main.onStart github.com/gravitational/teleport/tool/tbot/main.go:153 main.Run github.com/gravitational/teleport/tool/tbot/main.go:48 main.main runtime/proc.go:250 runtime.main runtime/asm_amd64.s:1594 runtime.goexit User Message: rpc error: code = Unknown desc = rpc error: code = InvalidArgument desc = missing parameter HostID] auth/register.go:590 INFO [AUTH] Attempting registration with auth server. auth/register.go:245 WARN [AUTH] Joining cluster without validating the identity of the Auth Server. This may open you up to a Man-In-The-Middle (MITM) attack if an attacker can gain privileged network access. To remedy this, use the CA pin value provided when join token was generated to validate the identity of the Auth Server. auth/register.go:411 INFO [AUTH] Attempting to register Bot with IAM method using regional STS endpoint auth/register.go:565 INFO [AUTH] Failed to register Bot using regional STS endpoint error:[ ERROR REPORT: Original Error: *status.Error rpc error: code = Unknown desc = unexpected HTTP status code received from server: 302 (Found); malformed header: missing HTTP content-type Stack Trace: github.com/gravitational/teleport/api@v0.0.0/client/joinservice.go:66 github.com/gravitational/teleport/api/client.(*JoinServiceClient).RegisterUsingIAMMethod github.com/gravitational/teleport/lib/auth/register.go:567 github.com/gravitational/teleport/lib/auth.registerUsingIAMMethod github.com/gravitational/teleport/lib/auth/register.go:348 github.com/gravitational/teleport/lib/auth.registerThroughAuth github.com/gravitational/teleport/lib/auth/register.go:246 github.com/gravitational/teleport/lib/auth.Register github.com/gravitational/teleport/lib/tbot/renew.go:477 github.com/gravitational/teleport/lib/tbot.(*Bot).getIdentityFromToken github.com/gravitational/teleport/lib/tbot/tbot.go:357 github.com/gravitational/teleport/lib/tbot.(*Bot).initialize github.com/gravitational/teleport/lib/tbot/tbot.go:228 github.com/gravitational/teleport/lib/tbot.(*Bot).Run github.com/gravitational/teleport/tool/tbot/main.go:234 main.onStart github.com/gravitational/teleport/tool/tbot/main.go:153 main.Run github.com/gravitational/teleport/tool/tbot/main.go:48 main.main runtime/proc.go:250 runtime.main runtime/asm_amd64.s:1594 runtime.goexit User Message: rpc error: code = Unknown desc = unexpected HTTP status code received from server: 302 (Found); malformed header: missing HTTP content-type] auth/register.go:590 INFO [AUTH] Attempting to register Bot with IAM method using global STS endpoint auth/register.go:565 INFO [AUTH] Failed to register Bot using global STS endpoint error:[ ERROR REPORT: Original Error: *status.Error rpc error: code = Unknown desc = unexpected HTTP status code received from server: 302 (Found); malformed header: missing HTTP content-type Stack Trace: github.com/gravitational/teleport/api@v0.0.0/client/joinservice.go:66 github.com/gravitational/teleport/api/client.(*JoinServiceClient).RegisterUsingIAMMethod github.com/gravitational/teleport/lib/auth/register.go:567 github.com/gravitational/teleport/lib/auth.registerUsingIAMMethod github.com/gravitational/teleport/lib/auth/register.go:348 github.com/gravitational/teleport/lib/auth.registerThroughAuth github.com/gravitational/teleport/lib/auth/register.go:246 github.com/gravitational/teleport/lib/auth.Register github.com/gravitational/teleport/lib/tbot/renew.go:477 github.com/gravitational/teleport/lib/tbot.(*Bot).getIdentityFromToken github.com/gravitational/teleport/lib/tbot/tbot.go:357 github.com/gravitational/teleport/lib/tbot.(*Bot).initialize github.com/gravitational/teleport/lib/tbot/tbot.go:228 github.com/gravitational/teleport/lib/tbot.(*Bot).Run github.com/gravitational/teleport/tool/tbot/main.go:234 main.onStart github.com/gravitational/teleport/tool/tbot/main.go:153 main.Run github.com/gravitational/teleport/tool/tbot/main.go:48 main.main runtime/proc.go:250 runtime.main runtime/asm_amd64.s:1594 runtime.goexit User Message: rpc error: code = Unknown desc = unexpected HTTP status code received from server: 302 (Found); malformed header: missing HTTP content-type] auth/register.go:590 ERROR: rpc error: code = Unknown desc = rpc error: code = InvalidArgument desc = missing parameter HostID, rpc error: code = Unknown desc = rpc error: code = InvalidArgument desc = missing parameter HostID, rpc error: code = Unknown desc = unexpected HTTP status code received from server: 302 (Found); malformed header: missing HTTP content-type, rpc error: code = Unknown desc = unexpected HTTP status code received from server: 302 (Found); malformed header: missing HTTP content-type ```

tbot attempts to connect. Looking at a packet capture, I can see it trying to use a custom ALPN dial method in the Client Hello.

snippets from wireshark packet capture ``` Handshake Protocol: Client Hello Handshake Type: Client Hello (1) ... Extension: server_name (len=25) Type: server_name (0) Length: 25 Server Name Indication extension Server Name list length: 23 Server Name Type: host_name (0) Server Name length: 20 Server Name: teleport.example.com ... Extension: application_layer_protocol_negotiation (len=25) Type: application_layer_protocol_negotiation (16) Length: 25 ALPN Extension Length: 23 ALPN Protocol ALPN string length: 19 ALPN Next Protocol: teleport-proxy-grpc ALPN string length: 2 ALPN Next Protocol: h2 ```

This explains why an ALB would return HTTP 464 since teleport-proxy-grpc is not an ALPN protocol that the alb understands.

Since my test was on a non-alb setup, I was able to see that tbot was still not able to join. It goes on to make another TLS connection. This time it sends server-name of teleport.cluster.local and alpn h2, which seems correct. Unfortunately, it gets a 302 response, which usually indicates that the web port is issuing a redirect to /web/login.

The Teleport code indicates that IAM joining requires a grpc client, which I believe should be feasible with tls multiplexing disabled.

Bug details:

stacktrace including 464 error ``` INFO [AUTH] Failed to register Bot using regional STS endpoint error:[ ERROR REPORT: Original Error: *status.Error rpc error: code = Unknown desc = unexpected HTTP status code received from server: 464 (); malformed header: missing HTTP content-type Stack Trace: github.com/gravitational/teleport/api@v0.0.0/client/joinservice.go:66 github.com/gravitational/teleport/api/client.(*JoinServiceClient).RegisterUsingIAMMethod github.com/gravitational/teleport/lib/auth/register.go:567 github.com/gravitational/teleport/lib/auth.registerUsingIAMMethod github.com/gravitational/teleport/lib/auth/register.go:287 github.com/gravitational/teleport/lib/auth.registerThroughProxy github.com/gravitational/teleport/lib/auth/register.go:246 github.com/gravitational/teleport/lib/auth.Register github.com/gravitational/teleport/lib/tbot/renew.go:477 github.com/gravitational/teleport/lib/tbot.(*Bot).getIdentityFromToken github.com/gravitational/teleport/lib/tbot/tbot.go:357 github.com/gravitational/teleport/lib/tbot.(*Bot).initialize github.com/gravitational/teleport/lib/tbot/tbot.go:228 github.com/gravitational/teleport/lib/tbot.(*Bot).Run github.com/gravitational/teleport/tool/tbot/main.go:234 main.onStart github.com/gravitational/teleport/tool/tbot/main.go:153 main.Run github.com/gravitational/teleport/tool/tbot/main.go:48 main.main runtime/proc.go:250 runtime.main runtime/asm_amd64.s:1594 runtime.goexit User Message: rpc error: code = Unknown desc = unexpected HTTP status code received from server: 464 (); malformed header: missing HTTP content-type] auth/register.go:590 INFO [AUTH] Attempting to register Bot with IAM method using global STS endpoint auth/register.go:565 INFO [AUTH] Failed to register Bot using global STS endpoint error:[ ERROR REPORT: Original Error: *status.Error rpc error: code = Unknown desc = unexpected HTTP status code received from server: 464 (); malformed header: missing HTTP content-type Stack Trace: github.com/gravitational/teleport/api@v0.0.0/client/joinservice.go:66 github.com/gravitational/teleport/api/client.(*JoinServiceClient).RegisterUsingIAMMethod github.com/gravitational/teleport/lib/auth/register.go:567 github.com/gravitational/teleport/lib/auth.registerUsingIAMMethod github.com/gravitational/teleport/lib/auth/register.go:287 github.com/gravitational/teleport/lib/auth.registerThroughProxy github.com/gravitational/teleport/lib/auth/register.go:246 github.com/gravitational/teleport/lib/auth.Register github.com/gravitational/teleport/lib/tbot/renew.go:477 github.com/gravitational/teleport/lib/tbot.(*Bot).getIdentityFromToken github.com/gravitational/teleport/lib/tbot/tbot.go:357 github.com/gravitational/teleport/lib/tbot.(*Bot).initialize github.com/gravitational/teleport/lib/tbot/tbot.go:228 github.com/gravitational/teleport/lib/tbot.(*Bot).Run github.com/gravitational/teleport/tool/tbot/main.go:234 main.onStart github.com/gravitational/teleport/tool/tbot/main.go:153 main.Run github.com/gravitational/teleport/tool/tbot/main.go:48 main.main runtime/proc.go:250 runtime.main runtime/asm_amd64.s:1594 runtime.goexit User Message: rpc error: code = Unknown desc = unexpected HTTP status code received from server: 464 (); malformed header: missing HTTP content-type] auth/register.go:590 INFO [AUTH] Attempting registration with auth server. auth/register.go:245 ERROR: rpc error: code = Unknown desc = unexpected HTTP status code received from server: 464 (); malformed header: missing HTTP content-type, rpc error: code = Unknown desc = unexpected HTTP status code received from server: 464 (); malformed header: missing HTTP content-type, rpc error: code = Unknown desc = unexpected HTTP status code received from server: 464 (); malformed header: missing HTTP content-type ```
strideynet commented 1 year ago

Is this a duplicate of https://github.com/gravitational/teleport/issues/20792 ?

webvictim commented 1 year ago

I believe the underlying issue is ~the same~ similar, but the issue's description makes it sound like this is a Machine ID-specific issue rather than something that applies to Teleport joining as a whole - @programmerq?

strideynet commented 1 year ago

I believe the underlying issue is ~the same~ similar, but the issue's description makes it sound like this is a Machine ID-specific issue rather than something that applies to Teleport joining as a whole - @programmerq?

Most of the code behind node and bot joining is shared, so I'd hazard a guess that they are potentially caused by the same problem.

strideynet commented 9 months ago

Potentially closed by https://github.com/gravitational/teleport/pull/23974

strideynet commented 9 months ago

Marking as resolved by https://github.com/gravitational/teleport/pull/23974