Webhook tries to add initContainer during UPDATE

[alculquicondor]

Describe the bug

Somehow, the vault-injector missed a Pod with a vault.hashicorp.com/agent-inject: true during CREATE, that is, this Pod doesn't have the vault initContainer. However, this is not the problem.

The problem is that the webhook is now trying to add the initContainer during Pod UPDATEs, which is not allowed by the k8s API. If this Pod belongs to a Job, it means that the k8s control plane cannot remove the tracking finalizer, with 2 consequences:

• The Job cannot be marked as finished.
• The Pod cannot be garbage collected.

To Reproduce

I don't know how the cluster that I investigated got into this situation, but I imagine that these steps should reproduce the problem:

1. Create a Job where the Pod template does NOT have the annotation vault.hashicorp.com/agent-inject: true.
2. Edit the Pod created to add the annotation vault.hashicorp.com/agent-inject: true.
3. Attempt to delete the Pod, which will never succeed.

Expected behavior

The vault injector should not try to add the initContainer during UPDATE, as this is disallowed by the API and it doesn't work under any circumstances.

Environment

• Kubernetes version: 1.26 or newer
  • Distribution or cloud vendor (OpenShift, EKS, GKE, AKS, etc.): any.
  • Other configuration options or runtime services (istio, etc.):
• vault-k8s version: I saw this in a very old version, feel free to close if this was already fixed.

Additional context Add any other context about the problem here.

[tomhjp]

Thanks for the report, I think this was fixed in v1.3.0 with https://github.com/hashicorp/vault-k8s/commit/adb76bff82bcdfc6d9361f7177b56f1a4436d58e. It sounds like you were probably on an older version than that if it was very old?

[alculquicondor]

Not my cluster, unfortunately, but this was detected by our monitoring system.

[alculquicondor]

/close

Let's hope the SEO directs affected users to this ticket :)

[alculquicondor]

In case you are interested, I'm also adding this to the k8s documentation: https://github.com/kubernetes/website/pull/43773