search

tumblr / k8s-sidecar-injector

Kubernetes sidecar injection service
Apache License 2.0
345 stars 75 forks source link

POD not getting sidecar even though injection is requested #61

Open borajuanjo opened 2 years ago

borajuanjo commented 2 years ago

What's going on?

PODs are not getting sidecars, even though the injection is requested. Funnily I got it to work once on a training cluster, and then I deleted the cluster and I can't get it to work again.

Expected Behavior

PODs should get created with sidecar.

Reproducer

I just went through the documentation step by step, and I can't identify what I'm doing wrong, or what I did differently that one time I got it to work.

Here are some logs for when the debian-debug POD gets deployed, but no sidecar.

10.64.4.1 - - [23/Mar/2022:15:16:18 +0000] "GET /health HTTP/2.0" 200 12 "" "kube-probe/1.21"
I0323 15:16:21.450826       1 webhook.go:510] AdmissionReview for Kind=/v1, Kind=Pod, Namespace=default Name= () UID=37184454-a6e7-4f35-be04-8eeaedf85265 patchOperation=CREATE UserInfo={system:serviceaccount:kube-system:replicaset-controller 4b17f21d-590c-4d37-acf2-5096af5e70cd [system:serviceaccounts system:serviceaccounts:kube-system system:authenticated] map[]}
I0323 15:16:21.450862       1 webhook.go:174] Pod / annotation injector.tumblr.com/request=test1 requesting sidecar config test1:latest
I0323 15:16:21.450961       1 webhook.go:548] AdmissionResponse: patch=[{"op":"add","path":"/spec/containers/0/env","value":[{"name":"HELLO","value":"world"}]},{"op":"add","path":"/spec/containers/0/env/-","value":{"name":"TEST","value":"test_that"}},{"op":"add","path":"/spec/containers/0/volumeMounts/-","value":{"name":"test-vol","mountPath":"/tmp/test"}},{"op":"add","path":"/spec/containers/-","value":{"name":"sidecar-nginx","image":"nginx:1.12.2","ports":[{"containerPort":80}],"env":[{"name":"ENV_IN_SIDECAR","value":"test-in-sidecar"},{"name":"HELLO","value":"world"},{"name":"TEST","value":"test_that"}],"resources":{},"volumeMounts":[{"name":"test-vol","mountPath":"/tmp/test"}],"imagePullPolicy":"IfNotPresent"}},{"op":"add","path":"/spec/volumes/-","value":{"name":"test-vol","configMap":{"name":"test-config"}}},{"op":"add","path":"/metadata/annotations/injector.tumblr.com~1status","value":"injected"}]
I0323 15:16:21.451004       1 webhook.go:626] Ready to write reponse ...
10.64.3.7 - - [23/Mar/2022:15:16:21 +0000] "POST /mutate?timeout=10s HTTP/1.1" 200 1237 "" "kube-apiserver-admission"
10.64.4.1 - - [23/Mar/2022:15:16:28 +0000] "GET /health HTTP/2.0" 200 12 "" "kube-probe/1.21"

Here's the debian-debug POD with no sidecar.

$ kubectl get po | grep debian
debian-debug                                             1/1     Running   0          55m

Version Deets

wiebeck commented 2 years ago

Very same here. Running on Azure Kubernetes with K8s v1.23.5

karolmalyszko commented 2 years ago

Got the same issue on AWS EKS v1.21, but made it to work. Looks like main problem lies in MutatingWebhookConfiguration. In original example manifests, webhook config had

which was refused by K8s in my case, had to modify it to

To conform with K8s requirements for this API version, I had to add additional keys for sideEffects: and admissionReviewVersions: (K8s docs here).

Two WAG's followed :: first one for sideEffects where i put 'None', second one for admissionReviewVersions where i started with 'v1'. This led to successful deployment of MutatingWebhookConfiguration, but the sidecars were not being added to test pods / deployments, same as for both of you. Search through codebase got me to webhook.go line 613 :: admissionReview := v1beta1.AdmissionReview{}

Change of admissionReviewVersions from v1 to v1beta1 got the problem solved, sidecar started as expected. Here's a gist with example mutating-webhook-configuration.yaml

Hope this helps.

mindovermiles262 commented 11 months ago

So helpful! Thank you! I also had this issue.

I've taken your image updates and pushed it to docker hub if anyone else would like to use it. mindovermiles262/sidecar-injector

Change the deployment.yaml image to:

spec:
  template:
    spec:
      containers:
      - name: "k8s-sidecar-injector"
        [ ... ]
        # image: tumblr/k8s-sidecar-injector:latest   <-- Comment this line out, and update
        image: mindovermiles262/sidecar-injector

Note, this image works with the updated mutating webhook configuration where the apiVersion: admissionregistration.k8s.io/v1 and webhooks.sideEffects: None