Mutating webhook breaks sidecars

Anomander commented 9 months ago

What happened: With the release of 1.29 the new Sidecars feature is available. The feature introduces the new field in the container definition that can mark some init containers as non-blocking and running perpetually.

The mutating webhook removes the new field from containers, preventing the new feature from being used.

What you expected to happen: Sidecars are updated with IRSA, but remain functional

How to reproduce it (as minimally and precisely as possible):

apiVersion: v1
kind: Namespace
metadata:
  name: dummy
  labels:
    team: dummy
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: dummy
  namespace: dummy
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::251000242201:role/CustomTestRole
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: dummy
  namespace: dummy
  labels:
    hive-app: dummy
spec:
  replicas: 1
  selector:
    matchLabels:
      hive-app: dummy
  template:
    metadata:
      name: dummy
      namespace: dummy
      labels:
        hive-app: dummy
    spec:
      serviceAccount: dummy
      initContainers:
      - name: sidecar-test-irsa
        image: ubuntu:latest
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
          limits:
            cpu: 120m
            memory: 120Mi
        command: ["/bin/sh"]
        args: ["-c", "
              apt update -q;
              apt install -q -y netcat;
              nc -lk 10000;
              while true; do sleep 100; done
              "]
        restartPolicy: Always
        imagePullPolicy: IfNotPresent
        ports:
      containers:
      - name: ubuntu
        image: ubuntu:latest
        resources:
          requests:
            cpu: 100m
            memory: 100Mi
          limits:
            cpu: 120m
            memory: 120Mi
        command: ["/bin/sh"]
        args: ["-c", "
              apt update -q;
              apt install -q -y netcat;
              nc -lk 10000;
              while true; do sleep 100; done
              "]
        imagePullPolicy: IfNotPresent
        ports:
      restartPolicy: Always

kubectl -n dummy get pods
NAME                     READY   STATUS     RESTARTS   AGE
dummy-6565945869-tdkgh   0/1     Init:0/1   0          7m26s

Containers:
  ubuntu:
...
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False

Anything else we need to know?: Here's the log of the mutation being applied, note the absence of the restartPolicy in the patch.

{
    "configuration": "pod-identity-webhook",
    "webhook": "iam-for-pods.amazonaws.com",
    "patch": [
        {
            "op": "add",
            "path": "/spec/volumes/0",
            "value": {
                "name": "aws-iam-token",
                "projected": {
                    "sources": [
                        {
                            "serviceAccountToken": {
                                "audience": "sts.amazonaws.com",
                                "expirationSeconds": 86400,
                                "path": "token"
                            }
                        }
                    ]
                }
            }
        },
        {
            "op": "add",
            "path": "/spec/containers",
            "value": [
                {
                    "name": "ubuntu",
                    "image": "ubuntu:latest",
                    "command": [
                        "/bin/sh"
                    ],
                    "args": [
                        "-c",
                        " apt update -q; apt install -q -y netcat; nc -lk 10000; while true; do sleep 100; done "
                    ],
                    "env": [
                        {
                            "name": "AWS_STS_REGIONAL_ENDPOINTS",
                            "value": "regional"
                        },
                        {
                            "name": "AWS_DEFAULT_REGION",
                            "value": "us-west-2"
                        },
                        {
                            "name": "AWS_REGION",
                            "value": "us-west-2"
                        },
                        {
                            "name": "AWS_ROLE_ARN",
                            "value": "arn:aws:iam::251000242201:role/CustomTestRole"
                        },
                        {
                            "name": "AWS_WEB_IDENTITY_TOKEN_FILE",
                            "value": "/var/run/secrets/eks.amazonaws.com/serviceaccount/token"
                        }
                    ],
                    "resources": {
                        "limits": {
                            "cpu": "120m",
                            "memory": "120Mi"
                        },
                        "requests": {
                            "cpu": "100m",
                            "memory": "100Mi"
                        }
                    },
                    "volumeMounts": [
                        {
                            "name": "kube-api-access-v5sdg",
                            "readOnly": true,
                            "mountPath": "/var/run/secrets/kubernetes.io/serviceaccount"
                        },
                        {
                            "name": "aws-iam-token",
                            "readOnly": true,
                            "mountPath": "/var/run/secrets/eks.amazonaws.com/serviceaccount"
                        }
                    ],
                    "terminationMessagePath": "/dev/termination-log",
                    "terminationMessagePolicy": "File",
                    "imagePullPolicy": "IfNotPresent"
                }
            ]
        },
        {
            "op": "add",
            "path": "/spec/initContainers",
            "value": [
                {
                    "name": "sidecar-test-irsa",
                    "image": "ubuntu:latest",
                    "command": [
                        "/bin/sh"
                    ],
                    "args": [
                        "-c",
                        " apt update -q; apt install -q -y netcat; nc -lk 10000; while true; do sleep 100; done "
                    ],
                    "env": [
                        {
                            "name": "AWS_STS_REGIONAL_ENDPOINTS",
                            "value": "regional"
                        },
                        {
                            "name": "AWS_DEFAULT_REGION",
                            "value": "us-west-2"
                        },
                        {
                            "name": "AWS_REGION",
                            "value": "us-west-2"
                        },
                        {
                            "name": "AWS_ROLE_ARN",
                            "value": "arn:aws:iam::251000242201:role/CustomTestRole"
                        },
                        {
                            "name": "AWS_WEB_IDENTITY_TOKEN_FILE",
                            "value": "/var/run/secrets/eks.amazonaws.com/serviceaccount/token"
                        }
                    ],
                    "resources": {
                        "limits": {
                            "cpu": "120m",
                            "memory": "120Mi"
                        },
                        "requests": {
                            "cpu": "100m",
                            "memory": "100Mi"
                        }
                    },
                    "volumeMounts": [
                        {
                            "name": "kube-api-access-v5sdg",
                            "readOnly": true,
                            "mountPath": "/var/run/secrets/kubernetes.io/serviceaccount"
                        },
                        {
                            "name": "aws-iam-token",
                            "readOnly": true,
                            "mountPath": "/var/run/secrets/eks.amazonaws.com/serviceaccount"
                        }
                    ],
                    "terminationMessagePath": "/dev/termination-log",
                    "terminationMessagePolicy": "File",
                    "imagePullPolicy": "IfNotPresent"
                }
            ]
        }
    ],
    "patchType": "JSONPatch"
}

Environment:

AWS Region: any
EKS Platform version (if using EKS, run aws eks describe-cluster --name <name> --query cluster.platformVersion): "eks.1"
Kubernetes version (if using EKS, run aws eks describe-cluster --name <name> --query cluster.version): "1.29"
Webhook Version: ?

DanielCKennedy commented 9 months ago

This is due to pod-identity-webhook using an outdated k8s client before the restartPolicy field was added to Containers type.

Have a fix here: https://github.com/aws/amazon-eks-pod-identity-webhook/pull/209

Eurymone commented 9 months ago

I investigated the issue as well, and came to the same conclusion regarding the source of the bug and fix to bring. We opened a ticket on AWS support side, and suggested it to bring it up here. It's currently making the SideCar feature unusable and it would be great to have that added to EKS asap 🙏

avisaradir commented 9 months ago

Is there an expected date and time for the distribution of this solution to #209

Im on the latest version of eks on both the cluster and nodes and this is still happening.

prateekgogia commented 9 months ago

Change is available in EKS, marking this as closed.

itayvolo commented 1 week ago

@prateekgogia To which version I need to upgrade in-order to have the fix? Currently experiencing this issue with k8s 1.29 with eks.17

aws / amazon-eks-pod-identity-webhook

Mutating webhook breaks sidecars #208