search

gravitational / teleport

The easiest, and most secure way to access and protect all of your infrastructure.
https://goteleport.com
GNU Affero General Public License v3.0
17.44k stars 1.74k forks source link

Bad Request | Teleport 11.3.5 | AWS Console #22471

Closed itsmesuniljacob closed 1 year ago

itsmesuniljacob commented 1 year ago

Expected behavior:

Trying to access AWS Management Console through Teleport. The AWS console should be accessed as a federated login

Current behavior: Getting Bad request

image

However AWS CLI is working perfectly without any issues Bug details:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "TeleportRole",
            "Effect": "Allow",
            "Action": [
                "sts:AssumeRole"
            ],
            "Resource": "arn:aws:iam::XXX:role/TeleportReadOnly"
        }
    ]
}
2023-02-27T10:46:33Z WARN [APP:SERVI] Failed to serve request: non-200 response from AWS federation endpoint: "400 Bad Request" map[Content-Security-Policy:[default-src 'none' https://aws.amazon.com https://*.signin.aws.amazon.com https://signin.aws.amazon.com https://*.analytics.console.aws.a2z.com  'nonce-dX+EemT/qICK3/BasPX7qg=='; script-src 'self'

AWS CLI access (i.e. tsh aws) is working fine The issue is with AWS web console access through Teleport.

greedy52 commented 1 year ago

First thanks for the debugging call.

I am able to reproduce it now with kube2iamand have found the actual root cause today. In short, the way kube2iam hijacks the ec2 metadata apis makes Teleport Agent thinks its host is an ec2, not a "temporary credential".

It's very hard to detect if kiam or kube2iam is hijacking the traffic. One possible workaround is to add an environment variable (e.g. TELEPORT_AWS_FEDERATION_TEMPORARY_CREDENTIALS=true) to force the decision on whether we think the host session is using temporary credentials. This could help debug future potential problems too.

The reason we haven't seen this before is we usually attach the IAM role either through IAM service account though odic or directly through the node's identity. And those are also valid options to avoid the kube2iam issue.

itsmesuniljacob commented 1 year ago

Thanks @greedy52. I was thinking that kube2iam intercepts traffic from the containers to the EC2 Metadata API, calling the AWS Security Token Service (STS) API to obtain temporary credentials using the pod configured role, then using these temporary credentials to perform the original request.

So, does this mean session duration may not be the culprit here?

This environment variable, where do we add the same?

greedy52 commented 1 year ago

So, does this mean session duration may not be the culprit here?

Correct. It's a red herring. The actual culprit is Teleport think it's using EC2/node credentials, instead of a temporary one.

This environment variable, where do we add the same?

We have to implement this toggle through an env var, and release it as a patch.

itsmesuniljacob commented 1 year ago

Thanks @greedy52. I tried with IRSA and I was able to access AWS console and AWS CLI, successfully. However, it seems I have an issues when doing kubectl operations , which is giving me 403 in the audit logs.

{
  "addr.local": "172.20.0.1:443",
  "addr.remote": "127.0.0.1:58240",
  "cluster_name": "teleport.example.com",
  "code": "T3009I",
  "ei": 0,
  "event": "kube.request",
  "kubernetes_cluster": "teleport.example.com",
  "kubernetes_groups": [
    "system:authenticated",
    "system:masters"
  ],
  "kubernetes_users": [
    "sunilj"
  ],
  "login": "sunilj",
  "namespace": "default",
  "proto": "kube",
  "request_path": "/api/v1/namespaces/default/pods",
  "resource_api_group": "core/v1",
  "resource_kind": "pods",
  "resource_namespace": "default",
  "response_code": 403,
  "server_id": "29bb5071-54b0-4924-bae3-ea3b2d173bde",
  "time": "2023-03-09T11:10:44.675Z",
  "uid": "e4c50f24-8596-447b-bd34-c1aed9401e0f",
  "user": "sunilj",
  "verb": "GET"
}
error: the server doesn't have a resource type "pod"
greedy52 commented 1 year ago

I tried with IRSA and I was able to access AWS console and AWS CLI, successfully.

Awesome! The web console session is likely limited to one hour for IRSA.

However, it seems I have an issues when doing kubectl operations , which is giving me 403 in the audit logs.

I am not a Kube Access expert, but I would first double-check if Kubernetes groups from tsh status is valid.. https://goteleport.com/docs/kubernetes-access/controls/

itsmesuniljacob commented 1 year ago

@greedy52 The Kubernetes groups show systems:masters

The logins show Logins: -teleport-nologin-e2008735-b66a-4b57-9e2e-77f0fa9b6605, -teleport-internal-join

greedy52 commented 1 year ago

The Logins are usernames for SSH access so I don't think it's related.

itsmesuniljacob commented 1 year ago

@greedy52 I have fixed the issue. It was the problem with cluster role binding. The service account name was not correct in Subjects field. This occurred, because I did manual changes

Really appreciate your help