Deleting policyServer.imagePullSecret puts kubewarden resource versions out of sync

[kravciak]

Current Behavior

Problem was found in lines 21 & 22 https://github.com/kubewarden/kubewarden-end-to-end-tests/blob/f89beb63db59574aba670148c688474edf887998/tests/private-registry-tests.bats#L20

When I

• set up defaults with secret helm update kubewarden-defaults --set policyServer.imagePullSecret=secret-registry-docker
  • deploy policy
  • delete secret
  • delete policy

Then resource versions go out of sync

~ kubectl get deploy -n kubewarden policy-server-default -o json
"kubewarden/config-version": "2275"

~ kubectl get cm -n kubewarden policy-server-default -o json | jq '.metadata.resourceVersion'
2459

All new policies stay in pending at this point because of ConfigurationVersionMismatch defined here - https://github.com/kubewarden/kubewarden-controller/blob/1c8afcce20f11f1de5056b8adcda53b51e455b3d/controllers/policystatus_utils.go#L49

It does not seem right it breaks this way, but maybe it's a feature - pls confirm...

To reproduce:

• comment out teardown_file function from private-registry-tests.bats, then

  
  make clean cluster install private-registry-tests.bats
  kubectl --namespace kubewarden delete secret secret-registry-docker
  kubectl delete clusteradmissionpolicies private-pod-privileged

check

kubectl get deploy -n kubewarden policy-server-default -o json kubectl get cm -n kubewarden policy-server-default -o json | jq '.metadata.resourceVersion'

[fabriziosestito]

Controller logs after deleting the secret:

2023-09-19T09:44:03+02:00       ERROR   Reconciler error        {"controller": "policyserver", "controllerGroup": "policies.kubewarden.io", "controllerKind": "PolicyServer", "PolicyServer": {"name":"default"}, "namespace": "", "name": "default", "reconcileID": "2ee43ac8-5aa1-4421-875b-c91fbb052df5", "error": "reconciliation error: cannot get spec.ImagePullSecret: secrets \"secret-registry-docker\" not found"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /home/lain/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:329
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /home/lain/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /home/lain/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:235

[fabriziosestito]

A solution could be skipping returning the error and just logging it. In this scenario the reconciliation loop will succeed but the new policyserver will be unable to mount the volume. Maybe this is clearer for the user:

Events:
  Type     Reason       Age                  From               Message
  ----     ------       ----                 ----               -------
  Normal   Scheduled    3m35s                default-scheduler  Successfully assigned kubewarden/policy-server-default-6956dcfcd6-xzb2c to k3d-kubewarden-testing-server-0
  Warning  FailedMount  93s                  kubelet            Unable to attach or mount volumes: unmounted volumes=[imagepullsecret], unattached volumes=[policy-store imagepullsecret sources kube-api-access-nlhkj certs policies]: timed out waiting for the condition
  Warning  FailedMount  88s (x9 over 3m36s)  kubelet            MountVolume.SetUp failed for volume "imagepullsecret" : secret "secret-registry-docker" not found