PostDelete Hook with other resources than pod/job (like SA)

[dtrouillet]

Checklist:

• [x] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [x] I've included steps to reproduce the bug.
• [x] I've pasted the output of argocd version.

Describe the bug

When I create a PostDelete hook with other resources like ServiceAccount, Role, RoleBinding, even if the hook succeed, the Application stay in Deleted state.

To Reproduce

Create resources like ServiceAccount, Role, RoleBinding and Pod/Job that use this ServiceAccount to works.

You can view this repo : https://github.com/dtrouillet/postdelete-hook-issue2

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: test2-damien
  namespace: argocd
spec:
  destination:
    namespace: test2-damien
    server: https://XXXXX
  project: default
  revisionHistoryLimit: 10
  sources:
    - path: .
      repoURL: https://github.com/dtrouillet/postdelete-hook-issue2.git
      targetRevision: master
  syncPolicy:
    syncOptions:
      - CreateNamespace=true

Expected behavior

Application is deleted after hook execution

Screenshots

Version

argocd: v2.10.0+2175939 BuildDate: 2024-02-06T14:28:43Z GitCommit: https://github.com/argoproj/argo-cd/commit/2175939ed6156ddd743e60f427f7f48118c971bf GitTreeState: clean GoVersion: go1.21.3 Compiler: gc Platform: linux/amd64

Logs

No logs

[dtrouillet]

Related to https://github.com/argoproj/argo-cd/pull/16595

[jgwest]

@dtrouillet Can you describe specifically what you expect will happen in this case? Are you saying A) the Application should be deleted, but isn't? Or B) are you saying the Application is deleted, but shouldn't be?

From the docs:

"Using a PostDelete hook to run clean-up or finalizer logic after all Application resources are deleted. Please note that PostDelete hooks are only deleted if the delete policy matches the aggregated deletion hooks status and not garbage collected after the application is deleted. "

My read of the dos is that the PostDelete hook does not prevent the deletion of the Application. AFAICT It is primarily used for post-delete notifications.

[dtrouillet]

@jgwest I expect the application to be deleted after the Hook (A). But in my case (the bug in question), my application is not deleted. It remains in "deleting", the hook was executed successfully (pod completed).

I was expecting the hook to be deleted as well as the application.

For information, I use this hook to annotate the namespace with a view to destroying it later (to compensate for the lack of "DeleteNamespace")

[joebowbeer]

IIUC, the Application is deleted before the post-delete hooks run.

But the Application will hang around until the post-delete finalizers are removed - which happens after success?

I think all the resource hooks are handled individually, and so the problem is likely that your service account or role or role binding was deleted as soon as it was applied (HookSuccess).

I suggest using something like ArgoWorkflows to run multi-resource resource hooks.

I suspect this bug should be closed as Won't Fix or Not Planned.

[sprat]

I have the same problem. My PostDelete hook pod needs a specific service account in order to operate. But as soon as I add the PostDelete annotation to the ServiceAccount, Role and RoleBinding, the Application refuse to delete, even if the post delete job succeeds. I added the argocd.argoproj.io/hook-delete-policy: HookSucceeded to all the PostDelete resources to make sure they are deleted after the phase completes, but this annotation seems to be ignored in this case. Note that the deletion works perfectly fine when there's only a job with a PostDelete annotation, so I would qualify this as a bug. Note also that there's no workaround, if we don't add the argocd.argoproj.io/hook: PostDelete annotation to the SA, R and RB, those resources are deleted before the PostDelete hook pod starts, so it does not work either.

I've noticed that the hook-delete-policy applies when the sync finish. For example, if you have PreSync and PostSync hooks with argocd.argoproj.io/hook-delete-policy: HookSucceeded, the hook resources are deleted after the sync terminates, i.e. after the PostSync pods terminates. That's probably why the deletion does not work here. I would have expected that the delete policy of each resource was applied after each "phase" completes (PreSync, PostSync, PostDelete).

[llavaud]

same problem here, my PostDelete hook needs a secret, if I create the secret without the PostDelete annotation the pod fails to start because the secret is already deleted, and with the annotation the pod succeed but the App stay in Deleting state forever...

[andrii-korotkov-verkada]

ArgoCD versions 2.10 and below have reached EOL. Can you upgrade and let us know if the issue is still present, please?

[sprat]

I have seen the same problem with ArgoCD 2.12.3, so I guess the problem is still there.