Describe the bug
The latest released version of the controller (v2.4.7), is still using the k8s.io/api dependency from Kubernetes 1.21, so, if you are running the controller on a newer Kubernetes version, the aws-load-balancer-webhook MutatingWebhookConfiguration add for the controller will remove any unknown field from the pods spec, essentially breaking the pod spec in some cases.
In my case I'm specially interested on start defining a gRPC liveness probe, which was only introduced on Kubernetes 1.23, however if the aws-load-balancer-controller is installed in the cluster, the cluster is receiving a mutating webhook for the newer created pod and removing the grpc section of the pod spec.
Steps to reproduce
Install the aws-load-balancer-controller version v2.4.7 on EKS cluster running Kubernetes 1.24.
The deployment will be created successfully, however no pod will be created out of the deployment, if you list the events in the namespace, you will find the following:
Error creating: Pod "podinfo-9b75f867-nxd5l" is invalid: spec.containers[1].readinessProbe: Required value: must specify a handler type
Expected outcome
I would expect the pod to be created out of the deployment, containing the grpc liveness and readiness probe as defined in the deployment.
Environment
AWS Load Balancer controller version: v2.4.7
Kubernetes version: 1.24
Using EKS (yes/no), if so version? yes, 1.24
Additional Context:
I can see in the main branch the go dependencies were already updated to match the Kubernetes 1.26 API version (#2998), when can we expect that a new version of the controller will be cut from the main branch? I will be happy to help on anything you need, like contributing with more integration tests and running a night build of the main branch internally and validate if it controller is working as expected.
Similar issue is also being discussed here https://github.com/aws/containers-roadmap/issues/1952, in that thread the suspicion was about the managed amazon-eks-pod-identity-webhook being the culprit, however I've ruled that out, in my case, if I try to create a pod using a gRPC probe on a new or upgraded EKS cluster running 1.24, the pod is created fine if I don't have the aws-load-balancer-controller installed in the cluster, once I get the controller installed, creating such a pod stop working.
Describe the bug The latest released version of the controller (
v2.4.7
), is still using thek8s.io/api
dependency from Kubernetes 1.21, so, if you are running the controller on a newer Kubernetes version, theaws-load-balancer-webhook
MutatingWebhookConfiguration add for the controller will remove any unknown field from the pods spec, essentially breaking the pod spec in some cases.In my case I'm specially interested on start defining a gRPC liveness probe, which was only introduced on Kubernetes 1.23, however if the aws-load-balancer-controller is installed in the cluster, the cluster is receiving a mutating webhook for the newer created pod and removing the
grpc
section of the pod spec.Steps to reproduce
Expected outcome I would expect the pod to be created out of the deployment, containing the
grpc
liveness and readiness probe as defined in the deployment.Environment
v2.4.7
1.24
1.24
Additional Context: I can see in the
main
branch the go dependencies were already updated to match the Kubernetes 1.26 API version (#2998), when can we expect that a new version of the controller will be cut from themain
branch? I will be happy to help on anything you need, like contributing with more integration tests and running a night build of the main branch internally and validate if it controller is working as expected.Similar issue is also being discussed here https://github.com/aws/containers-roadmap/issues/1952, in that thread the suspicion was about the managed
amazon-eks-pod-identity-webhook
being the culprit, however I've ruled that out, in my case, if I try to create a pod using a gRPC probe on a new or upgraded EKS cluster running 1.24, the pod is created fine if I don't have theaws-load-balancer-controller
installed in the cluster, once I get the controller installed, creating such a pod stop working.