open-policy-agent / gatekeeper

🐊 Gatekeeper - Policy Controller for Kubernetes
https://open-policy-agent.github.io/gatekeeper/
Apache License 2.0
3.73k stars 765 forks source link

Gatekeeper MutatingAdmissionWebhook conflicts with vpc-cni #3719

Open tomerleib opened 3 days ago

tomerleib commented 3 days ago

What steps did you take and what happened: We have deployed Gatekeeper on our EKS cluster and created assign to inject a new sidecar for pods in one of the namespaces. When a new pod that matched the assign was created, the sidecar was injected but VPC-CNI failed to assign an IP for this pod.

The error on the pod was Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "d456c3f7056f7574708efdcb161e676949989c33411c13eba09bb78502a9a7c3": plugin type="aws-cni" name="aws-cni" failed (add): add cmd: failed to assign an IP address to container.

After further investigation with AWS support, we discovered that there's a conflict between our vpc-cni that uses ANNOTATE_POD_IP and the gatekeeper webhook.

A similar issue can be found here: https://github.com/aws/amazon-vpc-cni-k8s/issues/2654#issuecomment-1811336162

What did you expect to happen: I was expecting the pod to start properly with network as everything else.

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

Environment: