search

canonical / admission-webhook-operator

Admission Webhook Operator
Apache License 2.0
1 stars 4 forks source link

`MutatingWebhookConfiguration` conflict when upgrading from 1.6 to 1.7 #88

Closed NohaIhab closed 1 year ago

NohaIhab commented 1 year ago

the name of mutatingWebhookConfiguration resource that the admission-webhook charm creates has been changed from admission-webhook in 1.6 to admission-webhook-mutating-webhook-configuration in 1.7. In the case of upgrading CKF 1.6 to 1.7, this leads to 2 mutatingWebhookConfiguration objects existing in the cluster for the same webhook. As a result, when creating a notebook, the webhook gets triggered twice from both mutatingWebhookConfiguration objects and the notebook gets stuck in FailedCreate with a conflict on the PodDefaults being applied to the notebook pod.

NohaIhab commented 1 year ago

Logs from the admission-webhook workload:

I0614 17:08:21.895892       1 main.go:660] 1 matching pod defaults, for pod testnb-0
I0614 17:08:21.895904       1 main.go:666] Matching PD detected of count 1, patching spec
I0614 17:08:21.895970       1 main.go:479] mutating pod: testnb-0
I0614 17:08:21.896013       1 main.go:681] applied poddefaults: access-ml-pipeline successfully on Pod: testnb-0 
I0614 17:08:21.904028       1 main.go:598] Entering mutatePods in mutating webhook
I0614 17:08:21.904575       1 main.go:624] Looking at pod annotations, found: map[kubectl.kubernetes.io/default-container:testnb kubectl.kubernetes.io/default-logs-container:testnb poddefault.admission.kubeflow.org/poddefault-access-ml-pipeline:34168 prometheus.io/path:/stats/prometheus prometheus.io/port:15020 prometheus.io/scrape:true sidecar.istio.io/status:{"initContainers":["istio-init"],"containers":["istio-proxy"],"volumes":["workload-socket","credential-socket","workload-certs","istio-envoy","istio-data","istio-podinfo","istio-token","istiod-ca-cert"],"imagePullSecrets":null,"revision":"default"}]
I0614 17:08:22.765280       1 main.go:644] fetched 1 poddefault(s) in namespace test
I0614 17:08:22.765324       1 main.go:660] 1 matching pod defaults, for pod testnb-0
I0614 17:08:22.765332       1 main.go:666] Matching PD detected of count 1, patching spec
E0614 17:08:22.765533       1 main.go:349] merging volumes for access-ml-pipeline has a conflict on volume-kf-pipeline-token:
NohaIhab commented 1 year ago

MutatingWebhookConfiguration objects in the cluster:

kubectl get mutatingwebhookconfigurations | grep admission-webhook
NAME                                                        WEBHOOKS   
kubeflow-admission-webhook                                  1          
kubeflow-admission-webhook-mutating-webhook-configuration   1