Importing eks cluster creates unauthorized kubeconfig connection secrets

[blut]

What happened?

When importing eks cluster into a managed resource using external-name, the created connection secret is created with an unauthorized kubeconfig (exceptions below). This is particularly unfortunate if the cluster is imported into a composite resource which uses provider-helm or provider-kubernetes resources.

From my understanding the issue is that GetConnectionConfig (https://github.com/crossplane/provider-aws/blob/e81c88171e567cb47faf3d40492ee63993518412/pkg/clients/eks/eks.go#L345 ) uses the caller sts token from the previous DescribeCluster call in the generated kubeconfig. This only works if the caller of the DescribeCluster is listed as cluster creator.

Normally, one can add cluster access through the aws-auth configmap, but even this configuration is not sufficient. Through using aws-auth, the caller can fetch a valid token trough a GetToken call, which would be a viable workaround.

How can we reproduce it?

1. create an eks cluster with one IAM user
2. import the cluster into a managed resource with external-name using a different IAM User or Role
3. fetch and test the kubeconfig

What environment did it happen in?

Crossplane version: 1.5.0 provider-aws version: 0.20.1

[haarchri]

hi @blut hope you're doing well

tested with:

Crossplane version: v1.5.1
provider-aws version: v0.22.0

we (@haarchri / @dwerder) did today the same and changed our setup backed by iam-user to a irsa-role - with the same effect you reported:

 Warning  CannotConnectToProvider  29m (x9 over 96m)  managed/release.helm.crossplane.io  cannot create new Kubernetes client: cannot create Kubernetes client: Unauthorized

NAME                                                             CHART        VERSION   SYNCED   READY   STATE   REVISION   DESCRIPTION   AGE
release.helm.crossplane.io/central-crossplane-prod-fpdpz-f8pkk   example-1   0.0.1     False                                             26h
release.helm.crossplane.io/central-crossplane-prod-fpdpz-jnhwg   example-2   0.0.1     False                                             26h
release.helm.crossplane.io/central-crossplane-prod-fpdpz-xc8bc   example-3   0.0.1     False                                             26h

then we investigated the provider-aws implementation against aws cli: getToken

kubectl get secrets 774643bf-996f-40b5-b3bb-9c04e50e8abd-eks --template={{.data.kubeconfig}} -n crossplane-system | base64 -D | grep "    token: k8s-aws-v1." | cut -c23- | base64 -D

https://sts.eu-central-1.amazonaws.com/?Action=GetCallerIdentity&
Version=2011-06-15&
X-Amz-Algorithm=AWS4-HMAC-SHA256&
X-Amz-Credential=<credential>%2F20211230%2Feu-central-1%2Fsts%2Faws4_request&
X-Amz-Date=20211230T132853Z&
X-Amz-Expires=60&
X-Amz-Security-Token=<long-security-token>&
X-Amz-SignedHeaders=host%3Bx-k8s-aws-id&
X-Amz-Signature=<signature>

aws eks get-token --cluster-name central-crossplane-prod --region eu-central-1 | jq -r .status.token | cut -c12- | base64 -D

https://sts.eu-central-1.amazonaws.com/?Action=GetCallerIdentity&
Version=2011-06-15&
X-Amz-Algorithm=AWS4-HMAC-SHA256&
X-Amz-Credential=<credential>%2Feu-central-1%2Fsts%2Faws4_request&
X-Amz-Date=20211230T133831Z&
X-Amz-Expires=60&
X-Amz-SignedHeaders=host%3Bx-k8s-aws-id&
X-Amz-Signature=<signature>  

we think that X-Amz-Security-Token is injected because IRSA-Role STS:AssumeWebRole - so it looks like that both implementations looks the same but the IRSA Role gets the error Unauthorized

so we enabled the CloudWatch-Logs in EKS-Cluster for authenticator and found the following log-entry:

2021-12-30T13:48:48.227+01:00Copytime="2021-12-30T12:48:47Z" level=warning msg="access denied"
arn="arn:aws:iam::123456789112:role/central-crossplane-prod" client="127.0.0.1:43974" error="ARN is not mapped: 
arn:aws:iam::123456789112:role/central-crossplane-prod" method=POST path=/authenticate sts=sts.eu-central-1.amazonaws.com 

so it looks like that the IRSA-Role is not mapped in aws-auth configmap - we added a new entry for our IRSA-Role like:

                - groups:
                  - system:masters
                  rolearn: arn:aws:iam::123456789112:role/central-crossplane-prod
                  username: central-crossplane-prod

after that we get it working again:

NAME                                                             CHART        VERSION   SYNCED   READY   STATE      REVISION   DESCRIPTION        AGE
release.helm.crossplane.io/central-crossplane-prod-fpdpz-f8pkk   example-1   0.0.1     True     True    deployed   1          Install complete    27h
release.helm.crossplane.io/central-crossplane-prod-fpdpz-jnhwg   example-2   0.0.1     True     True    deployed   2          Upgrade complete   27h
release.helm.crossplane.io/central-crossplane-prod-fpdpz-xc8bc   example-3   0.0.1     True     True    deployed   4          Upgrade complete   27h

[haarchri]

@blut did you have a Chance to check again ?

[kferrone]

I addd the IRSA role to the aws-auth. I'm still getting the same issues. I guess I am in the same situation. I have a Composition which creates an EKS cluster. Then I migrated the yaml to another cluster which means I am basically importing a cluster using the external-name annotation.

All of the other AWS resources are working just fine. I also had a value in the path like /eks/, I already learned the hard way you can't have anything in the path except / if you are to add it to the aws-auth.

[haarchri]

We create before the new entry in aws-auth configmap before migrating and it fits

[kferrone]

It actually did work! For whatever reason the Releases were not changing, I added a label like foo: bar and it caused all of them to start becoming ready and true.

[haarchri]

@blut i will Close this issue because it is working if the new role is added before in aws-auth configmap

[blut]

@haarchri, wow, thank's guys. I think I've failed to understand the solution here for too long.

So eventhough the role arn is arn:aws:iam::012345678912:role/project/managed/tenant-provider-aws, the aws-auth configmap requires arn:aws:iam::012345678912:role/tenant-provider-aws.