opa-gatekeeper can not work properly with different mutatingWebhookName

[ugur99]

What steps did you take and what happened: We are trying to deploy opa-gatekeeper with a different mutatingWebhookName rather than the default one, but it is not working properly. The same problem occurs when upgrading the existing 'opa-gatekeeper' deployment. The logs are as follows:

gatekeeper-controller-manager-7bd5f4b96f-h8mdl manager {"level":"error","ts":1689339424.419973,"logger":"cert-rotation","msg":"secret is not well-formed, cannot update webhook configurations","error":"Cert secret is not well-formed, missing ca.crt","errorVerbose":"Cert secret is not well-formed, missing ca.crt\ngithub.com/open-policy-agent/cert-controller/pkg/rotator.buildArtifactsFromSecret\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/github.com/open-policy-agent/cert-controller/pkg/rotator/rotator.go:385\ngithub.com/open-policy-agent/cert-controller/pkg/rotator.(*ReconcileWH).Reconcile\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/github.com/open-policy-agent/cert-controller/pkg/rotator/rotator.go:646\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:235\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_arm64.s:1172","stacktrace":"github.com/open-policy-agent/cert-controller/pkg/rotator.(*ReconcileWH).Reconcile\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/github.com/open-policy-agent/cert-controller/pkg/rotator/rotator.go:648\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:235"}
gatekeeper-controller-manager-7bd5f4b96f-h8mdl manager {"level":"error","ts":1689339424.4349635,"logger":"cert-rotation","msg":"Webhook not found. Unable to update certificate.","name":"gatekeeper-mutating-webhook-configuration","gvk":"admissionregistration.k8s.io/v1, Kind=MutatingWebhookConfiguration","error":"MutatingWebhookConfiguration.admissionregistration.k8s.io \"gatekeeper-mutating-webhook-configuration\" not found","stacktrace":"github.com/open-policy-agent/cert-controller/pkg/rotator.(*ReconcileWH).ensureCerts\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/github.com/open-policy-agent/cert-controller/pkg/rotator/rotator.go:684\ngithub.com/open-policy-agent/cert-controller/pkg/rotator.(*ReconcileWH).Reconcile\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/github.com/open-policy-agent/cert-controller/pkg/rotator/rotator.go:653\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:235"}gatekeeper-controller-manager-7bd5f4b96f-h8mdl manager {"level":"error","ts":1689339424.4349635,"logger":"cert-rotation","msg":"Webhook not found. Unable to update certificate.","name":"gatekeeper-mutating-webhook-configuration","gvk":"admissionregistration.k8s.io/v1, Kind=MutatingWebhookConfiguration","error":"MutatingWebhookConfiguration.admissionregistration.k8s.io \"gatekeeper-mutating-webhook-configuration\" not found","stacktrace":"github.com/open-policy-agent/cert-controller/pkg/rotator.(*ReconcileWH).ensureCerts\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/github.com/open-policy-agent/cert-controller/pkg/rotator/rotator.go:684\ngithub.com/open-policy-agent/cert-controller/pkg/rotator.(*ReconcileWH).Reconcile\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/github.com/open-policy-agent/cert-controller/pkg/rotator/rotator.go:653\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:235"}
gatekeeper-controller-manager-7bd5f4b96f-h8mdl manager {"level":"error","ts":1689339424.5442774,"logger":"cert-rotation","msg":"could not refresh CA and server certs","error":"Operation cannot be fulfilled on secrets \"gatekeeper-webhook-server-cert\": the object has been modified; please apply your changes to the latest version and try again","stacktrace":"github.com/open-policy-agent/cert-controller/pkg/rotator.(*CertRotator).refreshCertIfNeeded.func1\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/github.com/open-policy-agent/cert-controller/pkg/rotator/rotator.go:225\nk8s.io/apimachinery/pkg/util/wait.ConditionFunc.WithContext.func1\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:222\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtectionWithContext\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:235\nk8s.io/apimachinery/pkg/util/wait.runConditionWithCrashProtection\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:228\nk8s.io/apimachinery/pkg/util/wait.ExponentialBackoff\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:423\ngithub.com/open-policy-agent/cert-controller/pkg/rotator.(*CertRotator).refreshCertIfNeeded\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/github.com/open-policy-agent/cert-controller/pkg/rotator/rotator.go:252\ngithub.com/open-policy-agent/cert-controller/pkg/rotator.(*CertRotator).Start\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/github.com/open-policy-agent/cert-controller/pkg/rotator/rotator.go:184\nsigs.k8s.io/controller-runtime/pkg/manager.(*runnableGroup).reconcile.func1\n\t/go/src/github.com/open-policy-agent/gatekeeper/vendor/sigs.k8s.io/controller-runtime/pkg/manager/runnable_group.go:219"}
gatekeeper-controller-manager-7bd5f4b96f-h8mdl manager {"level":"info","ts":1689339424.5658972,"logger":"cert-rotation","msg":"no cert refresh needed"}
gatekeeper-controller-manager-7bd5f4b96f-h8mdl manager {"level":"info","ts":1689339424.6666622,"logger":"controller-runtime.healthz","msg":"healthz check failed","statuses":[{}]}
...
gatekeeper-controller-manager-7bd5f4b96f-h8mdl manager 2023/07/14 13:17:51 http: TLS handshake error from 192.168.49.2:5068: remote error: tls: bad certificate

Using the default value solves the problem. Could it be hard-coded somewhere in the code?

What did you expect to happen: mutatingWebhookName should be able to be manipulated for new and existing deployments.

Anything else you would like to add: Mutating webhooks are called in lexical order, and since we want to use opa-gatekeeper as the last mutator; we need to manipulate its mutationing webhook resource name something like zzz-gatekeeper-mutating-webhook-configuration.

Environment:

• Gatekeeper version: helm chart versions: 3.12.0 and latest
• Kubernetes version: (use kubectl version): 1.24.10

[acpana]

@ugur99 how are you trying to change the name? Are you using the flag? mutating-webhook-configuration-name

[ugur99]

Hi @acpana thank you for the quick reply, we're using helm chart in order to deploy and upgrade the gatekeeper. And related variable in the helm chart is mutatingWebhookName

[ugur99]

@acpana I think we found the problem, since the gatekeeper-controller-manager manifest is missing the appropriate flags, it uses the default values of these variables. This PR should fix the problem.