deleting an application that uses a certificate generated by cert-manager can lead to being stucked

[gigi206]

Checklist:

• [x] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [x] I've included steps to reproduce the bug.
• [x] I've pasted the output of argocd version.

Describe the bug If you install an application that requires a certificate generated by cert-manager and after you delete it on the web UI, you have a chance to be stuck for a long time because cert-manager tries to recreate immediately the certificate

To Reproduce

First, install cert-manager:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: cert-manager
  namespace: argo-cd
spec:
  destination:
    name: ''
    namespace: cert-manager-system
    server: 'https://kubernetes.default.svc'
  source:
    path: ''
    repoURL: 'https://charts.jetstack.io'
    targetRevision: v1.6.1
    chart: cert-manager
    helm:
      parameters:
        - name: installCRDs
          value: "true"
  project: default
  syncPolicy:
    syncOptions:
      - CreateNamespace=true
      - PruneLast=true

Create and apply the following file cluster-issuer.yaml:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: selfsigned-cluster-issuer
spec:
  selfSigned: {}

kubectl install -f cluster-issuer.yaml

Install an app that require a certificate:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: gitea
  namespace: argo-cd
spec:
  destination:
    namespace: gitea
    server: 'https://kubernetes.default.svc'
  source:
    chart: gitea
    repoURL: 'https://dl.gitea.io/charts'
    targetRevision: '5.0.1'
    helm:
      parameters:
        - name: gitea.admin.username
          value: demo
        - name: gitea.admin.password
          value: demo12345
        - name: ingress.enabled
          value: "true"
        - name: ingress.className
          value: nginx
        - name: ingress.hosts[0].host
          value: gitea.demo
        - name: ingress.hosts[0].paths[0].path
          value: /
        - name: ingress.hosts[0].paths[0].pathType
          value: Prefix
        - name: >-
            ingress.annotations.nginx\.ingress\.kubernetes\.io/server-alias
          value: gitea
        - name: ingress.tls[0].secretName
          value: gitea-cert-tls
        - name: ingress.tls[0].hosts[0]
          value: gitea.demo
        - name: >-
            ingress.annotations.cert-manager\.io/cluster-issuer
          value: selfsigned-cluster-issuer
  project: default
  syncPolicy:
    syncOptions:
      - CreateNamespace=true
      - PruneLast=true

Sometimes the application get stuck for a long time before completly deleting If I delete the namespace it solve this issue.

Expected behavior

When I click on the delete button, the application must be deleted

Version

argocd version
argocd: v2.2.2+03b17e0
  BuildDate: 2022-01-01T06:27:52Z
  GitCommit: 03b17e0233e64787ffb5fcf65c740cc2a20822ba
  GitTreeState: clean
  GoVersion: go1.16.11
  Compiler: gc
  Platform: linux/amd64

Logs Command argocd --port-forward --port-forward-namespace argo-cd --insecure app logs gitea show nothing

[TheDome]

Same for me. I am using LetsEncrypt ClusterIssuer to issue the certificates, but I am also receiving the following kubernetes logs:

develop   0s          Normal   CreateCertificate   ingress/dev-ingress                        Successfully created Certificate "dev-ingress-tls"

The current solution for me it to orphan delete the ingress manually, which is quite time consuming for several apps

[zswanson]

I think this may relate to #6873 I'm seeing this too, if we do a cascade delete of an ingress using certmanager tls, argo and cert-manager get stuck in a delete-create loop.

[Paragon1970]

Also seeing this behaviour with deletion of Argo applicationsets since under the hood it is removing an argo application. Currently on Argocd v2.2.5

[adiii717]

same here, the only thing that worked for me is to delete ingress manually.

[AndersBennedsgaard]

Cert-manager apparently does not work well with the default prune deletion propagation policy of foreground, as explained here. So setting

spec:
  syncPolicy:
    syncOptions:
    - PrunePropagationPolicy=background

in your Application manifest, makes the deletion much faster

[pdrastil]

Even when setting background propagation policy in Argo CD 2.5.7 the Certificate resource is still deleted with with foreground finalizer so above fix currently doesn't work.

[crenshaw-dev]

@pdrastil is that reproducible? If so, sounds like a bug in the application-controller. It should respect the configured propagation policy.

[pdrastil]

@crenshaw-dev I believe so - repro steps described in https://github.com/argoproj/argo-cd/issues/12453