Open rinswind opened 6 months ago
This could be useful. However, it doesn't necessarily require two separate tool chains. One can use the ECK ACK to create the PodIdentityAssociation and the IAM ACK to create the IAM role(s) and associated policies. This can all be accomplished in Kubernetes without any extra tooling like Terraform (other than the ACK controllers of course). There are other ways to do this as well.
Another argument for using annotations to create pod identity associations is that it is easier to trace the identities of a Pod. Go to the Pod, from there go to it's ServiceAccount, from there to it's IAM Role.
Also given that a pod identity association is meaningless without a ServiceAccount makes it more natural to create the former together with the latter.
These two arguments make the whole thing work in much more natural way rather than to have two separately managed resources.
I've gone the other way and enjoyed swapping from IRSA to PodIdentity so I can manage all the bindings as a platform engineer and my devs can either use them, or not use them on their own terms.
I really disliked having to own an IRSA binding in terraform, in addition to creating a k8s namespace, and service account, with the annotation, and then have the k8s resources in the terraform state - it blew up more than once on me. Then you need to make sure any helm installation uses the pre-created service account, and doesn't replace it (and a few did of popular third party helm charts).
All in all having these unlinked has been a huge blessing for me, and likewise to the point above I use Crossplane to handle creating PodIdentity XRDs from inside the internal helmcharts so it's all pretty seamless.
I prefer that the binding between IAM role and service account work in one direction or the other but not both.
With IRSA, it is the two-way binding between IAM role and service account annotation that confounds me.
@rinswind this article illustrates some ways to determine what is happening in a cluster:
https://securitylabs.datadoghq.com/articles/eks-pod-identity-deep-dive/
IIUC, the pod identity association is the last step in a chain of rules:
Is it correct to say that the distribution of responsibilities in your case goes like this:
If this is correct then there are some questions:
fyi code is here now - https://github.com/aws/eks-pod-identity-agent 🙏🏾
I really would like a way to do what GKE does where it is just based on namespace/service-account for all clusters. Having to create a per cluster binding is annoying, and I wish I have a way to create one association with the cluster portion being a glob.
Community Note
Tell us about your request In IRSA the mapping of a ServiceAccount to a IAM role is done by placing the annotation
eks.amazonaws.com/role-arn
on the ServiceAccount object. Similarly the pod identity agents (or a dedicated controller) should create Pod Identity Associations for the current EKS cluster when it discovers an annotation (possibly even the same annotation) on the ServiceAccount.Which service(s) is this request for? EKS
Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? It is hard to use Pod Identity when the deployment of a single application requires the carefully coordinated execution of two separate tool chains. For the EKS cluster we need to run Helm (e.g. via ArgoCD) and for the IAM roles that support the application we need to run something like Terraform. Given the tight coupling between a ServiceAccount and it's backing IAM Role it should be possible to provision both layers (application and infrastructure) together. This is similar to the tight coupling between a Service and it's backing Load Balancer.