bug: manually deleting an app from an appset does not trigger a recreate

[DButter]

Checklist:

• [X] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [X] I've included steps to reproduce the bug.
• [X] I've pasted the output of argocd version.

Describe the bug

Manually deleting a resource from an applicationset leads to a

applications.argoproj.io "XXX not found

Error, afterwhich the appset will not reconcile the state anymore

To Reproduce

create appset:

---
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: test
  namespace: argocd
spec:
  syncPolicy:
    preserveResourcesOnDeletion: false
  generators:
    - git:
        repoURL: 'ssh://git@git.git'
        revision: test
        directories:
          - path: resources/*
  template:
    metadata:
      name: 'test--{{path.basename}}-test'
    spec:
      project: test
      source:
        repoURL: 'ssh://git@git.git'
        targetRevision: test
        path: '{{path}}'
      destination:
        name: test
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
          allowEmpty: true

target-app:

any app yml

Expected behavior

Recreation of the app

Version

v2.10.1+a79e0ea

Logs

status:
  conditions:
    - lastTransitionTime: '2024-02-26T12:23:21Z'
      message: >-
        applications.argoproj.io
        "test" not
        found
      reason: UpdateApplicationError
      status: 'True'
      type: ErrorOccurred
    - lastTransitionTime: '2024-02-26T09:47:28Z'
      message: Successfully generated parameters for all Applications
      reason: ParametersGenerated
      status: 'True'
      type: ParametersGenerated
    - lastTransitionTime: '2024-02-26T12:23:21Z'
      message: >-
        applications.argoproj.io
        "test" not
        found
      reason: UpdateApplicationError
      status: 'False'
      type: ResourcesUpToDateype: ResourcesUpToDate

[bruno561]

In my case, after I rolled out HAProxy from Redis and (re)created the Application manually (kubectl apply -f), I had no more problems. Furthermore, this problem has already happened to me in versions prior to 2.10 and intermittently, that is, it already excludes Applications that were automatically recreated (as expected) and Applications that were not.

[DButter]

I'd rather know were the appset keeps its state so i can also delete the application there tbh

[JorTurFer]

We are facing too with this issue, and the workaround is restarting the appset-controller. It looks like if the controller stores a cached version that is tried to be validated on each loop. When we restart the controller that cached entry is gone and it works again

[DButter]

@JorTurFer Thanks for the workaround, seems to really be unintended behaviour if a restart fixes it.

[blakepettersson]

I've seen this in some environments, but have not been able to repro this locally. Any ideas? Which k8s versions are you all using? Which k8s flavor (EKS, AKS, GKE)?

[donovanrost]

I'm just now running into this on EKS