Install eks-pod-identity-webhook automatically for IRSA

[h3poteto]

/kind feature

Refs: https://github.com/kubernetes/kops/issues/8264

1. Describe IN DETAIL the feature/behavior/change you would like to see. At the moment, we can set up IRSA (IAM Roles for Service Accounts) using serviceAccountIssuerDiscovery and serviceAccountExternalPermissions. But we still must install eks-pod-identity-webhook to inject volume and env to pods. I hope that webhook will be installed automatically when I specify serviceAccountIssuerDiscovery in the cluster spec.

2. Feel free to provide a design supporting your feature request. In Makefile, https://github.com/aws/amazon-eks-pod-identity-webhook/blob/35a57cc479ae760760bfa9b5a628a488a46adad2/Makefile#L72-L88 it set CA_BUNDLE and approve CSR when I deploy eks-pod-identity-webhook.

But probably Kubernetes custom controller can execute these processes, so I tried to implement it on a trial basis. https://github.com/h3poteto/eks-pod-identity-webhook-installer

I think that kops can install it automatically. There are two methods:

1. Execute the same logic in kops-controller
2. Setup CRD of my repository and create custom resource

What do you think about this?

[olemarkus]

If you are willing to support the webhook as an addon, I support this.

Certificates can be handled by cert manager. We already use this for metrics-server and aws lbc.

The addon should not be installed when IRSA is enabled, but should require explicit config (e.g spec.podIdentityWebhook.enabled: true)

[h3poteto]

Thank you.

I also though about using cert manager for webhook certificates, but maybe it is difficult. Because eks-pod-identity-webhook doesn't accept custom cert and key when I use in-cluster config. https://github.com/aws/amazon-eks-pod-identity-webhook/blob/35a57cc479ae760760bfa9b5a628a488a46adad2/main.go#L153-L193

spec.podIdentityWebhook.enabled: true

That's nice.

[olemarkus]

From the looks of it, you just add the key/cert through cli flags...

[h3poteto]

If I set tls-key and tls-cert flags, I have to set kubeconfig too and I can't set in-cluster. So eks-pod-identity-webhook doesn't use in-cluster config as kubeconfig. Because when I set in-cluster flag, this block is executed. https://github.com/aws/amazon-eks-pod-identity-webhook/blob/35a57cc479ae760760bfa9b5a628a488a46adad2/main.go#L154-L186 In this block, tlsCertFile/tlsKeyFile are not read. To read its, I have to execute this block: https://github.com/aws/amazon-eks-pod-identity-webhook/blob/35a57cc479ae760760bfa9b5a628a488a46adad2/main.go#L188-L192 So I have to specify kubeconfig flag instead of in-cluster flag.

And actually I tried to specify tls-key and tls-cert on my cluster. But eks-pod-identity-webhook doesn't use its without kubeconfig flag.

[olemarkus]

Ah interesting design choice. I see there are some PRs and a lot of issues in this area ...

The only solution I can think of here is to make the webhook a daemonset and provision TLS and kubeconfig locally. but that is entirely overkill for something like this. One of the reasons I want to do a custom implementation of this, I guess.

[h3poteto]

Yes, if we provide kubeconfig for this, we can use cert-manager. Hmm... probably we can do a custom implementation to read tls-key and tls-cert without kubeconfig.

On the other hand, if we give up to use cert-manager for eks-pod-identity-webhook, we can deploy the webhook.

I'm wondering which on is better.

[h3poteto]

/assign

[h3poteto]

We can now use cert-manager to handle webhook certificates :laughing: https://github.com/aws/amazon-eks-pod-identity-webhook/pull/134

[olemarkus]

Oh wow. I guess this is worth considering again.

[h3poteto]

Yeah, I'm going to do this issue, and probably I can do this:

Certificates can be handled by cert manager. We already use this for metrics-server and aws lbc.