[EKS] [request]: Create Pod Identity Associations based on annotations on ServiceAccounts

[rinswind]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request In IRSA the mapping of a ServiceAccount to a IAM role is done by placing the annotation eks.amazonaws.com/role-arn on the ServiceAccount object. Similarly the pod identity agents (or a dedicated controller) should create Pod Identity Associations for the current EKS cluster when it discovers an annotation (possibly even the same annotation) on the ServiceAccount.

Which service(s) is this request for? EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? It is hard to use Pod Identity when the deployment of a single application requires the carefully coordinated execution of two separate tool chains. For the EKS cluster we need to run Helm (e.g. via ArgoCD) and for the IAM roles that support the application we need to run something like Terraform. Given the tight coupling between a ServiceAccount and it's backing IAM Role it should be possible to provision both layers (application and infrastructure) together. This is similar to the tight coupling between a Service and it's backing Load Balancer.

[joshuabaird]

This could be useful. However, it doesn't necessarily require two separate tool chains. One can use the ECK ACK to create the PodIdentityAssociation and the IAM ACK to create the IAM role(s) and associated policies. This can all be accomplished in Kubernetes without any extra tooling like Terraform (other than the ACK controllers of course). There are other ways to do this as well.

[rinswind]

Another argument for using annotations to create pod identity associations is that it is easier to trace the identities of a Pod. Go to the Pod, from there go to it's ServiceAccount, from there to it's IAM Role.

Also given that a pod identity association is meaningless without a ServiceAccount makes it more natural to create the former together with the latter.

These two arguments make the whole thing work in much more natural way rather than to have two separately managed resources.

[danielloader]

I've gone the other way and enjoyed swapping from IRSA to PodIdentity so I can manage all the bindings as a platform engineer and my devs can either use them, or not use them on their own terms.

I really disliked having to own an IRSA binding in terraform, in addition to creating a k8s namespace, and service account, with the annotation, and then have the k8s resources in the terraform state - it blew up more than once on me. Then you need to make sure any helm installation uses the pre-created service account, and doesn't replace it (and a few did of popular third party helm charts).

All in all having these unlinked has been a huge blessing for me, and likewise to the point above I use Crossplane to handle creating PodIdentity XRDs from inside the internal helmcharts so it's all pretty seamless.

[joebowbeer]

I prefer that the binding between IAM role and service account work in one direction or the other but not both.

With IRSA, it is the two-way binding between IAM role and service account annotation that confounds me.

@rinswind this article illustrates some ways to determine what is happening in a cluster:

https://securitylabs.datadoghq.com/articles/eks-pod-identity-deep-dive/

IIUC, the pod identity association is the last step in a chain of rules:

• IRSA: Looking for credentials via assume-role-with-web-identity
• ...
• Pod Identity: Looking for credentials via container-role

[rinswind]

@danielloader I am interested in other's experience with platform engineering.

Is it correct to say that the distribution of responsibilities in your case goes like this:

1. dev and ops agree on a service account #name, namespace and a IAM role that allows certain limited access to AWS
2. ops will create the IAM role and the Pod Identity Association the service account (should it appear later)
3. dev may choose later to deploy a pod and service account into the namespace and claim the IAM role for the pods

If this is correct then there are some questions:

• What are the security implications? You have a pod identity association that can accidentally be claimed by the wrong app
• How is step 1. conducted? Perhaps a platform service where devs make IAM access requests for a particular app and ops inspect the request and granting access?

[dims]

fyi code is here now - https://github.com/aws/eks-pod-identity-agent 🙏🏾

[wy100101]

I really would like a way to do what GKE does where it is just based on namespace/service-account for all clusters. Having to create a per cluster binding is annoying, and I wish I have a way to create one association with the cluster portion being a glob.