Mutating webhook is not compatible with Kubernetes >= 1.23

[rodrigorfk]

Describe the bug The latest released version of the controller (v2.4.7), is still using the k8s.io/api dependency from Kubernetes 1.21, so, if you are running the controller on a newer Kubernetes version, the aws-load-balancer-webhook MutatingWebhookConfiguration add for the controller will remove any unknown field from the pods spec, essentially breaking the pod spec in some cases.

In my case I'm specially interested on start defining a gRPC liveness probe, which was only introduced on Kubernetes 1.23, however if the aws-load-balancer-controller is installed in the cluster, the cluster is receiving a mutating webhook for the newer created pod and removing the grpc section of the pod spec.

Steps to reproduce

• Install the aws-load-balancer-controller version v2.4.7 on EKS cluster running Kubernetes 1.24.
• Create a pod as following:

  apiVersion: apps/v1
  kind: Deployment
  metadata:
  labels:
  app.kubernetes.io/name: podinfo
  name: podinfo
  spec:
  replicas: 1
  selector:
  matchLabels:
    app.kubernetes.io/name: podinfo
  template:
  metadata:
    labels:
      app.kubernetes.io/name: podinfo
  spec:
    containers:
    - name: podinfo
      ports:
      - containerPort: 7777
        name: grpc
        protocol: TCP
      image: ghcr.io/stefanprodan/podinfo:6.3.5
      imagePullPolicy: IfNotPresent
      command:
        - ./podinfo
        - --grpc-port
        - "7777"
      livenessProbe:
        grpc:
          port: 7777
      readinessProbe:
        grpc:
          port: 7777

• The deployment will be created successfully, however no pod will be created out of the deployment, if you list the events in the namespace, you will find the following:

  Error creating: Pod "podinfo-9b75f867-nxd5l" is invalid: spec.containers[1].readinessProbe: Required value: must specify a handler type

Expected outcome I would expect the pod to be created out of the deployment, containing the grpc liveness and readiness probe as defined in the deployment.

Environment

• AWS Load Balancer controller version: v2.4.7
• Kubernetes version: 1.24
• Using EKS (yes/no), if so version? yes, 1.24

Additional Context: I can see in the main branch the go dependencies were already updated to match the Kubernetes 1.26 API version (#2998), when can we expect that a new version of the controller will be cut from the main branch? I will be happy to help on anything you need, like contributing with more integration tests and running a night build of the main branch internally and validate if it controller is working as expected.

Similar issue is also being discussed here https://github.com/aws/containers-roadmap/issues/1952, in that thread the suspicion was about the managed amazon-eks-pod-identity-webhook being the culprit, however I've ruled that out, in my case, if I try to create a pod using a gRPC probe on a new or upgraded EKS cluster running 1.24, the pod is created fine if I don't have the aws-load-balancer-controller installed in the cluster, once I get the controller installed, creating such a pod stop working.

[kishorj]

/kind bug

@rodrigorfk, as you mentioned this will be fixed in the upcoming v2.5.0 release - we are close to the release.

[kishorj]

we released v.2.5.0, closing this issue.