Deadlock on deleting application

[ralf-cestusio]

During experiments i found the following "deadlock" when deleting an application.

I have an application which provisions an appproject a namespace and a new application (which is in the new project and deloys services in the new namespace)

Doing a cascading delete on test-app yields the following result:

the local-joe application stays behind with the error message: InvalidSpecError Application referencing project local-joe which does not exist

Application test-app stays behind with the error: DeletionError Operation cannot be fulfilled on namespaces "local-joe": The system is ensuring all content is removed from this namespace. Upon completion, this namespace will automatically be purged by the system.

What inspecting the cluster shows is that all the resources from app local joe are gone. The namespace is deleted

A way to fix this behavior is to manually issue:

kubectl delete apps local-joe
application.argoproj.io "local-joe" deleted

which removes the local-joe app and allows test-app to delete.

Also attached the application controller log. delete.log

[mvpmvh]

for some reason, kubectl delete apps <app_name> -n <namespace> did not work for me. The response application.argoproj.io "<app_name>" deleted looks like it worked, but if I kubectl get apps -n <namespace>, the app still remains.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[TomHellier]

for some reason, kubectl delete apps <app_name> -n <namespace> did not work for me. The response application.argoproj.io "<app_name>" deleted looks like it worked, but if I kubectl get apps -n <namespace>, the app still remains.

I had a finalizer in the app, going in and editing that to - finalizer: [] allowed it to delete.

[lechuk47]

Hi, I have the same problem with v1.3.0, until I delete the finalizer the Application object could not be deleted.

Any suggestion or information on this?

Regards

[alexec]

We could do with clear repro steps for this one.

[wdullaer]

I just had this with the latest argo. I'll see if I can reliably reproduce this.

I created the application by kubectl apply -f the manifest into the cluster. This had a finalizer specified in the metadata. If I removed the app with kubectl delete it would move the state to deleting, but not delete anything, not the app, nor any of the resources. I then edited the manifest and removed the finalizer. This caused argo to remove the app, but did not remove any of the resources. If I removed through the UI, the resources would be removed, but the app itself would not be removed. Again: once I removed the finalizer from the manifest using kubectl edit, the app would be removed as well.

This was on a clean ArgoCD install, no other apps in the k8s cluster. The only modifications I had done was load a custom tool using an init container and an update to the configmap to enable it.

[yebolenko]

@alexmt by the way: using the latest argocd, I have reproduced the same issue This time, when I manually remove the finalizer, application is not removed and I see that finalizer appears again and again inside of application definition

[aMiss-aWry]

for some reason, kubectl delete apps <app_name> -n <namespace> did not work for me. The response application.argoproj.io "<app_name>" deleted looks like it worked, but if I kubectl get apps -n <namespace>, the app still remains.

I'm running into this precise issue. How exactly are people resolving it? I've tried 'argocd app delete (app)', 'kubectl delete apps (app)', 'kubectl delete apps (app) --wait=False', but the apps continue sticking. I do not have a finalizer defined in the kubeconfig.

[srinath-panda]

same issue here.

[kksudo]

Remove finalizer in the all apps in the current namespace and delete:

for app in $(kubectl get applications  --output=jsonpath={.items..metadata.name}); do \
  echo $app && \
  kubectl patch app $app  --type json -p='[{"op": "remove", "path": "/metadata/finalizers"}]'; \
  kubectl delete $app; \
done

Remove kubernetes finalizer with kubectl cli

[ebuildy]

Just curious, why we always set the finalizer? (if you must remove it to delete the app)

[anthonyalayo]

I recently hit this, and I did find the reference in the docs: https://argo-cd.readthedocs.io/en/stable/user-guide/app_deletion/#deletion-using-kubectl

But I find this all a little silly like the comment above me hinted at.

[adiii717]

Remove finalizer in the all apps in the current namespace and delete:

for app in $(kubectl get applications  --output=jsonpath={.items..metadata.name}); do \
echo $app &&\
kubectl patch app $app  --type json -p='[{"op": "remove", "path": "/metadata/finalizers"}]'; \
kubectl delete $app; \
done

thanks, this help me to remove project

for proj in $(kubectl get appprojects -n argocd -o jsonpath={.items..metadata.name}); do \
    echo $proj &&\
    kubectl patch appproject $proj -n argocd --type json -p='[{"op": "remove", "path": "/metadata/finalizers"}]'; \
    kubectl delete appproject $proj -n argocd; \
done

[andrii-korotkov-verkada]

Do I understand correctly, that the example provided has a circular dependency of apps, projects and namespaces between each other? If so, this may just be an anti-pattern to avoid.