search

cybozu-go / accurate

Kubernetes controller for multi-tenancy. It propagates resources between namespaces accurately and allows tenant users to create/delete sub-namespaces.
https://cybozu-go.github.io/accurate/
Apache License 2.0
38 stars 5 forks source link

Should not attempt to finalize SubNamespace containing sub-namespaces #144

Closed erikgb closed 1 month ago

erikgb commented 1 month ago

Describe the bug

Today we experienced a strange issue in one of our clusters: It seems like a client managed to delete a SubNamespace that should have been blocked by the Accurate SubNamespace webhook - since the sub-namespace has child namespaces. This is probably not correct, but the result is that Accurate controller is trying to do something that is blocked by the Accurate Namespace webhook.

From controller logs:

{"level":"info","ts":"2024-07-16T12:05:08Z","msg":"starting finalization","controller":"subnamespace","controllerGroup":"accurate.cybozu.com","controllerKind":"SubNamespace","SubNamespace":{"name":"sam-review","namespace":"sam"},"namespace":"sam","name":"sam-review","reconcileID":"61bee8e0-880a-4717-88d3-13d6f8834620"}
{"level":"error","ts":"2024-07-16T12:05:08Z","msg":"Reconciler error","controller":"subnamespace","controllerGroup":"accurate.cybozu.com","controllerKind":"SubNamespace","SubNamespace":{"name":"sam-review","namespace":"sam"},"namespace":"sam","name":"sam-review","reconcileID":"61bee8e0-880a-4717-88d3-13d6f8834620","error":"failed to finalize: failed to delete namespace sam-review: admission webhook \"namespace.accurate.cybozu.io\" denied the request: child namespaces exist","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.0/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.0/pkg/internal/controller/controller.go:261\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.18.0/pkg/internal/controlle...

Environments

To Reproduce

Not sure if I know how to reproduce.

Expected behavior A clear and concise description of what you expected to happen.

The SubNamespace delete should have been blocked by the Accurate SubNamespace webhook. But when that is allowed go through (SubNamespace deletionTimestamp is set), the controller should not attempt a doomed operation.

Additional context Add any other context about the problem here.

erikgb commented 1 month ago

Another possible solution is to allow finalization, since there is no way to undelete a resource in Kubernetes, i.e. cascade delete ref. https://github.com/cybozu-go/accurate/issues/119. It should be possible to configure the Accurate webhooks to allow otherwise blocked deletes if the API call comes from Accurate or the Kubernetes garbage collector.