argoproj / argo-cd

Declarative Continuous Deployment for Kubernetes
https://argo-cd.readthedocs.io
Apache License 2.0
18.01k stars 5.49k forks source link

ClusterWorkflow Template no namespace application is causing thousands of orphaned instances #20430

Open keekdageek opened 1 month ago

keekdageek commented 1 month ago

Describe the bug

We are creating a Cluster Workflow Templates application a namespace. The application project includes the orphanedResources.warn = true which then leads to ArgoCD reporting pretty much every resource in the cluster orphaned and starts to slow down the performance.

I'm aware of the workaround to just remove the orphanedResources from the project which is what we are doing.

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: devops-ignore-orphan
  namespace: argo-system
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  description: "AppProject for managing devops instances of Applications that don't report orphaned"
  sourceRepos:
    - 'https://argoproj.github.io/argo-helm'
    - 'https://foobarcom/devops/cicd/devops-deployment-manifests'
    - '7777777777777.dkr.ecr.us-east-1.amazonaws.com/devops/cicd'
    - '7777777777777.dkr.ecr.us-west-2.amazonaws.com/devops/cicd'
  destinations:
    - namespace: "*"
      server: '*'
  clusterResourceWhitelist:
    - group: 'rbac.authorization.k8s.io'
      kind: ClusterRole
    - group: 'rbac.authorization.k8s.io'
      kind: ClusterRoleBinding
    - group: 'apiextensions.k8s.io'
      kind: CustomResourceDefinition
    - group: 'argoproj.io'
      kind: ClusterWorkflowTemplate
    - group: ''
      kind: Namespace

  orphanedResources:
    warn: true

It appears to be related to how belongToAnotherApp is calculated, in particular how the appKey := ctrl.toAppKey(appName) is calculated for this type of appproject without a namespace

https://github.com/argoproj/argo-cd/blob/9b11b21f00f006ec5bfca1ff39210e54b65bf4b5/controller/appcontroller.go#L600-L604

To Reproduce

Setup apprproject like defined above with a Cluster no namespace resource

Expected behavior

Don't show orphanedInstances seamlessly without the performance hit or check for this case and inform the user that it doesn't work with this type of application.

Version

❯ argocd version
argocd: v2.11.5+c4b283c
  BuildDate: 2024-07-15T18:15:32Z
  GitCommit: c4b283ce0c092aeda00c78ae7b3b2d3b28e7feec
  GitTreeState: clean
  GoVersion: go1.21.12
  Compiler: gc
  Platform: darwin/arm64
argocd-server: v2.11.5+c4b283c
  BuildDate: 2024-07-15T17:39:54Z
  GitCommit: c4b283ce0c092aeda00c78ae7b3b2d3b28e7feec
  GitTreeState: clean
  GoVersion: go1.21.10
  Compiler: gc
  Platform: linux/amd64
  Kustomize Version: v5.2.1 2023-10-19T20:13:51Z
  Helm Version: v3.14.4+g81c902a
  Kubectl Version: v0.26.11
  Jsonnet Version: v0.20.0
andrii-korotkov-verkada commented 2 weeks ago

AppKey includes the namespace https://github.com/argoproj/argo-cd/blob/9b11b21f00f006ec5bfca1ff39210e54b65bf4b5/controller/appcontroller.go#L2398

We are creating a Cluster Workflow Templates application a namespace.

Can you rephrase it, please? I'm having a hard time understanding what exactly is being done. It'd be helpful if you share some resources definitions and application manifest.

keekdageek commented 5 days ago

k, application set and example resource, all resources in this application would be the same cluster workflow templates which don't have a namespace.

appset

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: argo-workflows-cluster-wft
  namespace: argo-system
spec:
  goTemplate: true
  generators:
    - git:
        repoURL: https://gitlab.com/devops/cicd/devops-deployment-manifests
        revision: main
        directories:
          - path: applications/devops/argo-workflows-cluster-wft/*
          - exclude: true
            path: applications/devops/argo-workflows-cluster-wft/*use1
          - exclude: true
            path: applications/devops/argo-workflows-cluster-wft/*.disable*
  syncPolicy:
    preserveResourcesOnDeletion: false
  template:
    metadata:
      name: "argo-workflows-cluster-wft-{{ .path.basenameNormalized }}"
      labels:
        service-name: argo-workflows-cluster-wft
        deployment-type: system
    spec:
      project: devops-no-orphan
      source:
        repoURL: https://gitlab.com/devops/cicd/devops-deployment-manifests
        targetRevision: main
        path: "{{ .path.path }}"
      destination:
        name: '{{
          ternary
            "devops-doeks-sysopmaster-eks-cluster-v1"
            (printf "%s-eks-cluster-v1" (.path.basenameNormalized))
            ( eq .path.basenameNormalized "sysopmaster" )
          }}'
      syncPolicy:
        syncOptions:
          - ApplyOutOfSyncOnly=true
          - CreateNamespace=false

cluster WF template

apiVersion: argoproj.io/v1alpha1
kind: ClusterWorkflowTemplate
metadata:
  name: utils-cluster-wft
  labels:
    {{- include "common.labels" . |nindent 4 }}
spec:
  volumeClaimTemplates:
    - metadata:
        name: mnt-code
      spec:
        accessModes: ["ReadWriteOnce"]
        resources:
          requests:
            storage: 1Mi
        storageClassName: gp2
  volumes:
    - name: secrets
      csi:
        driver: secrets-store.csi.k8s.io
        readOnly: true
        volumeAttributes:
          secretProviderClass: "secrets"
  templates:
    # sh-mntvol - run sh command on image assume a mount valume.  Often used to list contents in a volume.
    - name: sh-mntvol
      inputs:
        parameters:
          - name: sh
          - name: image
            default: alpine
          - name: volume
            default: mnt-code
          - name: mountPath
            default: /mnt/code
      container:
        image: '{{ printf "{{inputs.parameters.image}}" }}'
        command: [sh, -c]
        args: ['{{ printf "{{inputs.parameters.sh}}" }}']
        volumeMounts:
          - name: '{{ printf "{{inputs.parameters.volume}}" }}'
            persistentVolumeClaim:
              claimName: '{{ printf "{{inputs.parameters.volume}}" }}'
            mountPath: '{{ printf "{{inputs.parameters.mountPath}}" }}'

    # echo - the input parameter surrounded but gt & lt signs ><
    - name: echo
      inputs:
        parameters:
          - name: text
      script:
        image: alpine
        command: ['/bin/sh']
        source: |
          echo '>{{ printf "{{inputs.parameters.text}}" }}<'

    # gitlab-code - clone a git repo, create config, commit, and mr json/yml files in <mountPath>/vars/git
    - name: gitlab-code
      inputs:
        parameters:
          - name: repo
          - name: branch
            default: main
          - name: volume
            default: mnt-code
          - name: mountPath
            default: /mnt/code
          - name: debug
            default: "false"
      script:
        # todo change image to gitlab-tools
        image: amazonaws.com/devops/cicd/docker/jinja-tools:0.2.0
        command: [ '/bin/sh' ]
        source: |
          set -e
          repo=$(basename '{{ printf "{{inputs.parameters.repo}}" }}' .git)
          git clone --depth 1 -b {{ printf "{{inputs.parameters.branch}}" }} https://${GIT_USERNAME}:${GIT_PASSWORD}@gitlab.retirementpartner.com/{{ printf "{{inputs.parameters.repo}}" }} {{ printf "{{inputs.parameters.mountPath}}" }}/$repo
          git_dir='{{ printf "{{inputs.parameters.mountPath}}" }}/vars/git'
          commit-dump $git_dir -g -a -r "{{ printf "{{inputs.parameters.mountPath}}" }}/$repo"
          if [[ '{{ printf "{{inputs.parameters.debug}}" }}' == "true" ]]; then
            cat $git_dir/*
          fi
        env:
          - name: GIT_USERNAME
            valueFrom:
              secretKeyRef:
                name: gitlab-creds
                key: gitlab_username
          - name: GIT_PASSWORD
            valueFrom:
              secretKeyRef:
                name: gitlab-creds
                key: gitlab_password
        volumeMounts:
          - name: '{{ printf "{{inputs.parameters.volume}}" }}'
            mountPath: '{{ printf "{{inputs.parameters.mountPath}}" }}'
          - name: 'secrets-gitlab'
            mountPath: '/mnt/secrets-gitlab'