Throw the error that could not update status when delete VaultSecret via foreground cascading deletion

[EricSyu]

I suffer an issue that could not delete VaultSecret successfully after execute delete VaultSecret via argocd.

I analyzed the behavior of argocd deletion. The default deletion method for argocd is to use foreground cascading deletion.

I try to reproduce the situation. I use kubectl to delete VaultSecret via foreground cascading deletion.

kubectl delete vaultsecret/helloworld --cascade='foreground'

I found that vault-secret-operator always outputs the following logs with an error on the last line.

{"level":"info","ts":"2023-07-10T14:54:26Z","msg":"Create client to get secret from Vault","controller":"vaultsecret","controllerGroup":"ricoberger.de","controllerKind":"VaultSecret","VaultSecret":{"name":"helloworld","namespace":"default"},"namespace":"default","name":"helloworld","reconcileID":"cb544d11-d78b-4398-94ee-dfcac26ab8e4","vaultRole":"vault-secrets-operator"}
{"level":"info","ts":"2023-07-10T14:54:26Z","logger":"vault","msg":"Read secret kvv2/helloworld"}
{"level":"info","ts":"2023-07-10T14:54:26Z","msg":"Creating a new Secret","controller":"vaultsecret","controllerGroup":"ricoberger.de","controllerKind":"VaultSecret","VaultSecret":{"name":"helloworld","namespace":"default"},"namespace":"default","name":"helloworld","reconcileID":"cb544d11-d78b-4398-94ee-dfcac26ab8e4","Secret.Namespace":"default","Secret.Name":"helloworld"}
{"level":"info","ts":"2023-07-10T14:54:26Z","msg":"Create client to get secret from Vault","controller":"vaultsecret","controllerGroup":"ricoberger.de","controllerKind":"VaultSecret","VaultSecret":{"name":"helloworld","namespace":"default"},"namespace":"default","name":"helloworld","reconcileID":"f8168091-2ce7-489c-b16a-93a82e5d92fd","vaultRole":"vault-secrets-operator"}
{"level":"info","ts":"2023-07-10T14:54:26Z","logger":"vault","msg":"Read secret kvv2/helloworld"}
{"level":"info","ts":"2023-07-10T14:54:26Z","msg":"Creating a new Secret","controller":"vaultsecret","controllerGroup":"ricoberger.de","controllerKind":"VaultSecret","VaultSecret":{"name":"helloworld","namespace":"default"},"namespace":"default","name":"helloworld","reconcileID":"f8168091-2ce7-489c-b16a-93a82e5d92fd","Secret.Namespace":"default","Secret.Name":"helloworld"}
{"level":"error","ts":"2023-07-10T14:54:26Z","msg":"Could not update status","controller":"vaultsecret","controllerGroup":"ricoberger.de","controllerKind":"VaultSecret","VaultSecret":{"name":"helloworld","namespace":"default"},"namespace":"default","name":"helloworld","reconcileID":"f8168091-2ce7-489c-b16a-93a82e5d92fd","error":"Operation cannot be fulfilled on vaultsecrets.ricoberger.de \"helloworld\": StorageError: invalid object, Code: 4, Key: /registry/ricoberger.de/vaultsecrets/default/helloworld, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 4d96a214-43b2-40fd-8aef-a87ae956d3f2, UID in object meta: ","stacktrace":"github.com/ricoberger/vault-secrets-operator/controllers.(*VaultSecretReconciler).updateConditions\n\t/workspace/controllers/vaultsecret_controller.go:238\ngithub.com/ricoberger/vault-secrets-operator/controllers.(*VaultSecretReconciler).Reconcile\n\t/workspace/controllers/vaultsecret_controller.go:181\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:235"}

According to the log logic, the VaultSecret is marked to be deleted, but vault-secret-operator still creates its secret and updates its status.

After VaultSecret is deleted by foreground cascading deletion, I guess there are two scenarios:

1. If VaultSecret has been removed by K8S GC, vault-secret-operator fails to update its status.(The log belongs to this case)
2. If VaultSecret has not yet been removed by K8S GC, vault-secret-operator creates new secret and update its status. Then it conflicts with foreground cascading deletion. Because in foreground cascading deletion, VaultSecret has to wait for the dependents to be removed before VaultSecret will be removed. This causes VaultSecret to be deleted unsuccessfully.

So I think the vault-secret-operator should not do anything after VaultSecret is deleted via foreground cascading deletion. The VaultSecret is just waiting to be removed by K8S GC.

[ricoberger]

Thanks for reporting and providing the fix 🙂