pod-identity-webhook deployed by EKS but API version not updated when upgrading to EKS 1.22

[j10may]

What happened: I am in the process of upgrading EKS clusters to 1.22. When checking for deprecated API versions in my clusters, the pod-identity-webhook is using the deprecated MutatingWebhookConfiguration api Version: "admissionregistration.k8s.io/v1beta1"

What you expected to happen:

From what I can see, the pod-identity-webhook was deployed when my EKS clusters were created, we have no configuration that manages this resource.

I assumed upgrading the EKS control plane to 1.22 would also update the API version of the pod-identity-webhook. However, when I upgraded my EKS cluster to 1.22, the pod-identity-webhook is still using "apiVersion":"admissionregistration.k8s.io/v1beta1"

Is this assumption correct - that EKS manages the pod-identity-webhook? or should I manage it as per the instructions in this repo?

How to reproduce it (as minimally and precisely as possible): Upgrade an EKS cluster from 1.21 to 1.22 and pod-identity-webhook api version remains the same

Anything else we need to know?:

Environment:

• AWS Region: Multiple but upgrade tested in ap-southeast-1
• EKS Platform version (if using EKS, run aws eks describe-cluster --name <name> --query cluster.platformVersion): eks.9
• Kubernetes version (if using EKS, run aws eks describe-cluster --name <name> --query cluster.version): 1.21 to 1.22
• Webhook Version: unknown, it was deployed by EKS control plane?

[Phil1602]

Hi,

how did you check for deprecated versions in your cluster?

I did the same last month ago and I used to use kube-no-trouble therefor and encountered the same "issue".

The pod-identity-webhook still got the version (admissionregistration.k8s.io/v1beta1) it was initially applied with written to the kubectl.kubernetes.io/last-applied-configuration annotation - that's probably the reason how it can be recognized as deprecated by tools like kube-no-trouble.

Due to the transparent API version conversion/representation (API Groups and versioning) everything still works as expected and shows admissionregistration.k8s.io/v1 after upgrading the control plane.

However the kubectl.kubernetes.io/last-applied-configuration annotation still shows admissionregistration.k8s.io/v1beta1, because it seems to not be re-applied.

For new clusters, initially deployed using >1.22, everything works as expected and it it initally applied using the right version.

[chickenbeef]

+1 to what @Phil1602 mentioned above. There's a known issue with kubent detecting the wrong apiVersion of the webhook.

You can verify it was updated by explicitly setting the version when running your kubectl command:

  $ kubectl get MutatingWebhookConfiguration.v1.admissionregistration.k8s.io pod-identity-webhook -o yaml | grep apiVersion
  apiVersion: admissionregistration.k8s.io/v1

v1beta1 should return no results:

  $ kubectl get MutatingWebhookConfiguration.v1beta1.admissionregistration.k8s.io pod-identity-webhook -o yaml | grep apiVersion
  error: the server doesn't have a resource type "MutatingWebhookConfiguration"

[j10may]

Thanks both for your answers - very helpful! I will run the commands suggested by @chickenbeef shortly. However, I deleted the pod-identity-webhook on my test cluster to see what would happen so I will need to run the upgrade on another cluster before I can confirm. I will let you know how it goes.

To answer @Phil1602's question:

how did you check for deprecated versions in your cluster?

I am using the pluto tool. I think this part in the README confused me to think the API server may not give the correct version but the linked issue mentions you can check with the specific version as @chickenbeef suggested.

You might think, "I'll just ask the api-server to tell me!", but this is fraught with danger. If you ask the api-server to give you deployments.v1.apps, and the deployment was deployed as deployments.v1beta1.extensions, the api-server will quite happily convert the api version and return a manifest with apps/v1. This is fairly well outlined in the discussion in https://github.com/kubernetes/kubernetes/issues/58131#issuecomment-356823588.

[mikkergimenez]

Does that mean we can/should ignore the fact that it was deployed with the old version? Could we manually apply deploy/mutatingwebhook.yaml to get kubent to be quiet or will it do something weird with the ca injector?

[j10may]

Happy New Year! I've tested on another cluster and I can confirm after the upgrade to 1.22 I get the following result:

kubectl get MutatingWebhookConfiguration.v1beta1.admissionregistration.k8s.io pod-identity-webhook -o yaml | grep apiVersion
Error from server (NotFound): the server could not find the requested resource

Whereas pre-upgrade this check returned an output.

Thanks for the help! I will close this issue

[rubenag97]

How do you solve it?