Custom application.instanceLabelKey delete app bug

ttk commented 3 years ago

Describe the bug

With the following argocd config: application.instanceLabelKey: argocd.argoproj.io/instance, argocd does not delete an app when the associated application.yaml (resource) is deleted from the repo. This it works correctly when a custom application.instanceLabelKey isn't applied.

To Reproduce

apply the config application.instanceLabelKey: argocd.argoproj.io/instance
deploy an app declaratively via the Application resource in a yaml file
delete the file corresponding to the App. Sync if required
observe that the app wasn't deleted

Expected behavior

after the Application.yaml file is deleted, the app should be deleted from argocd

Additional Observations/Notes

yes, the resources-finalizer.argocd.argoproj.io finalizer on the application is applied
if the app.kubernetes.io/instance label is manually edited and added to the Application resource (alongside the custom instance label), then the delete operation succeeds
manually running kubectl delete app APPNAME also successfully deletes the app

Version

argocd version
argocd: v1.8.4+28aea3d
  BuildDate: 2021-02-05T17:58:47Z
  GitCommit: 28aea3dfdede00443b52cc584814d80e8f896200
  GitTreeState: clean
  GoVersion: go1.14.12
  Compiler: gc
  Platform: windows/amd64
argocd-server: v1.8.4+28aea3d
  BuildDate: 2021-02-05T17:56:06Z
  GitCommit: 28aea3dfdede00443b52cc584814d80e8f896200
  GitTreeState: clean
  GoVersion: go1.14.12
  Compiler: gc
  Platform: linux/amd64
  Ksonnet Version: v0.13.1
  Kustomize Version: v3.9.2 2021-01-17T17:44:16Z
  Helm Version: v3.4.1+gc4e7485
  Kubectl Version: v1.17.8
  Jsonnet Version: v0.17.0

jannfis commented 3 years ago

I could not reproduce this behavior in v2.0.1.

I'll try to reproduce it with v1.8.7.

jannfis commented 3 years ago

Ah sorry, I have used the kubectl delete method, which you say works. Will try using delete from Git.

ttk commented 3 years ago

@jannfis Just wondering if you were able to reproduce this issue? I can do more debugging on my end if required.

ttk commented 3 years ago

I tested to see if the newer version of argocd (v2.0.4+0842d44) resolves this issue, and it does not. We will be upgrading to v2.1 soon, so I will test it again at that point. It would be great if someone else can confirm/reproduce this issue, that way I can rule out a problem with our specific install and configuration.

ttk commented 3 years ago

Tested again with v2.1.0+d0b2d55. Same problem.

AllexVeldman commented 2 years ago

@ttk I reproduced this using v2.1.7+a408e29 It seems to happen on any resource managed by ArgoCD, not just Application.

When the file is deleted from git the resource just disappears from the UI. Adding the app.kubernetes.io/instance label manually will cause the resource to show up in the UI as OutOfSync (requires pruning)

Maybe this will be solved with the annotation introduced in v2.2 as it would remove the need for using a custom instance label

ttk commented 2 years ago

I tested this again with (argocd v2.2.5+8f981cc) using the new resource tracking introduced in 2.2. I enabled it using application.resourceTrackingMethod: annotation config, then allowed ArgoCD to sync up the new tracking annotation on all the resources. However, the bug remains. If I delete the Application resource from git, ArgoCD doesn't detect it, and the app remains. I had to revert back to the original resource tracking method.

I would love to use the new resource tracking method as our teams have run into conflicts with the app.kubernetes.io/instance with various helm charts. However, the fact that resources aren't properly deleted when removed from git using the new resource tracking method is actually a worse bug than the helm chart conficts, so we are going with the lesser of the two evils.

pasha-codefresh commented 2 years ago

i will take a look @ttk

mmerickel commented 2 years ago

I'm seeing this issue as well. Specifically that when using application.resourceTrackingMethod: annotation and removing resources from the git config, they just disappear from argo-cd instead of showing as out-of-sync-waiting-to-be-pruned.

Using argo-cd 2.4.0.

todaywasawesome commented 2 years ago

This issue sees to persist, see #9522 which was marked duplicate in favor of this issue.

javajawa commented 2 months ago

Some 2024 updates:

2.11.3 shows the behaviour as described in this repo.
2.12.3 is showing new and subtly different behaviour, which might help debug what is going wrong

The test on 2.12.3 was performed using an app-of-apps structure, with a top level app I will call base. base's manifest was dozen Application resources, with their own manifests. Upon removing an application resource from base's manifest in git and running a sync, that Application resource is immediately removed from the cluster. However, it still appears in the top level application information as needing to be deleted (some disconnect between the various caches?).

The status message shows as OutOfSync (This resource is not present in the application's source. It will be deleted from Kubernetes if the prune option is enabled during sync.) (seems correct, although the resource has already been deleted).

Live manifest will show something similar to

Resource not found in cluster: undefined/undefined:keda-master-19ce710a
Please update your resource specification to use the latest Kubernetes API resources supported by the target cluster. The recommended syntax is argoproj.io/v1alpha1/Application:keda-master-19ce710a

which might imply it's losing track of the apiVersion and Kind somehow?

javajawa commented 2 months ago

I took a dive through the codebase to see if I could spot any odd placed a value might be hard coded, and didn't turn up anything -- excluding yaml, markdown, and test files, it only appears in two contexts in the code -- config.go where it definitely should be, and a load of doc strings for the CLI app.

I then attempted to try and follow the logic for where the various delete functions get called, and I think the next questions it would be prudent to look into are:

is it going through Delete at all;
is the Finalizer being set;
is the finalizer matching the right resources (I couldn't find the relevant code); and
what Delete Resource is called on

Sadly I'm fresh out of to-be-decommissioned clusters I'm able to break in the name of debugging, but if there's some test/debug builds in a few weeks time I'll be happy to give them a go

argoproj / argo-cd

Custom application.instanceLabelKey delete app bug #6061