ApplicationSet delete managedNamespaceMetadata when generate Application

[disaster37]

Checklist:

• [ ] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [ ] I've included steps to reproduce the bug.
• [ ] I've pasted the output of argocd version.

Describe the bug

ApplicationSet not set managedNamespaceMetadata section when generate application.

To Reproduce

Create applicationSet that contain on template the section managedNamespaceMetadata

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: test
spec:
  syncPolicy:
    preserveResourcesOnDeletion: true
  generators:
  - list:
      elements:
      - cluster: in-cluster
  template:
    metadata:
      name: 'test'
      annotations:
        argocd.argoproj.io/manifest-generate-paths: .
    spec:
      project: default
      destination:
        namespace: default
        name: '{{ cluster }}'
      source:
        repoURL: https://github.com/xxx.git
        path: 'test/'
        targetRevision: 'master'
      syncPolicy:
        managedNamespaceMetadata:
          labels:
            field.cattle.io/projectId: p-wd2xt
          annotations:
            field.cattle.io/projectId: c-57d7n:p-wd2xt
            field.cattle.io/resourceQuota: '{"limitsCpu":"8","limitsMemory":"8Gi","requestsStorage":"60Gi"}'
        automated:
          selfHeal: true
        syncOptions:
        - CreateNamespace=true
        - ApplyOutOfSyncOnly=true

It generate application with following manifest:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  annotations:
    argocd.argoproj.io/manifest-generate-paths: .
  creationTimestamp: '2022-09-21T08:20:03Z'
  generation: 92595
  name: test
  namespace: argocd
  ownerReferences:
    - apiVersion: argoproj.io/v1alpha1
      blockOwnerDeletion: true
      controller: true
      kind: ApplicationSet
      name: etloutils-activemq
      uid: 653f0ce6-293e-48ad-8129-158af507af65
  resourceVersion: '484324067'
  uid: b40f00ed-2068-44b5-98a9-0c1df19fcb42
spec:
  destination:
    name: in-cluster
    namespace: default
  project: default
  source:
    repoURL: https://github.com/xxx.git
    path: 'test/'
    targetRevision: 'master'
  syncPolicy:
    automated:
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
      - ApplyOutOfSyncOnly=true
status:
  ...

Like you can see, the managedNamespaceMetadata section is no more exist.

Expected behavior

It generate Application that contain section managedNamespaceMetadata

Screenshots

Version

ArgoCD 2.6.1

Logs

Paste any relevant application logs here.

[crenshaw-dev]

Is your Application CRD up to date?

[disaster37]

Yes I have loaded (kubectl replace) from 2.6.1 tag repository and /manifest/crds

I can create Application resources with managedNamespaceMetadata

[crenshaw-dev]

Hm. You've confirmed your ApplicationSet CRD is up to date as well?

[disaster37]

Hi, I have tested with ArgoCD 2.6.7, the matter.

My upgrade process:

1. Upgrade CRDs

   ```bash
   export KUBECONFIG=/home/user/.kube/config
   cd /tmp
   git clone https://github.com/argoproj/argo-cd.git -b v2.6.7
   cd argo-cd
   kubectl apply --server-side -k manifests/crds/

2. Upgrade ArgoCD via argocd-operator v0.5.0

   apiVersion: argoproj.io/v1alpha1
   kind: ArgoCD
   metadata:
   name: argocd
   labels:
   topology: ha
   spec:
   version: v2.6.7
   ...

[dmpe]

Yes, i can also confirm the same behavior on my end. Exactly as described, the generated application does not have managedNamespaceMetadata section, hence no labels/annotations are properly attached.

My CRDs are also up to date and we are using 2.7.7 argocd version.

[blakepettersson]

I can't reproduce this. The ApplicationSet YAML below

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: guestbook-test
  namespace: argocd
spec:
  generators:
    - clusters: {}
      selector:
        matchExpressions:
          - key: server
            operator: In
            values:
              - https://kubernetes.default.svc
  template:
    metadata:
      name: guestbook
    spec:
      project: default
      source:
        repoURL: https://github.com/argoproj/argocd-example-apps.git
        path: helm-dependency
        targetRevision: 'master'
        helm:
          valueFiles:
            - 'values.yaml'
      destination:
        namespace: guestbook
        server: https://kubernetes.default.svc
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
        managedNamespaceMetadata:
          labels:
            istio-injection: enabled
        syncOptions:
          - CreateNamespace=true

Gives this Application spec:

[dmpe]

I think my case is a bit different and that is likely the issue - I have following ApplicationSet which points to a gitlab repository.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: abc-application-set
spec:
  generators:
  - list:
      elements:
      - name: uat
        branch: branch_b
      - name: copy
        branch: branch_a
  template:
    metadata:
      name: 'abc-{{name}}-apps'
      labels:
        env: '{{name}}'
    spec:
      project: 'abc'
      source:
        repoURL: 'https://gitlab.firm.com/abc/argocd.git'
        path: 'lib/argocd/'
        targetRevision: '{{branch}}'
        directory:
          recurse: true
      destination:
        server: https://kubernetes.default.svc
        namespace: argocd
      syncPolicy:
        managedNamespaceMetadata:
          annotations:
            field.cattle.io/projectId: c-m-d658bggg:p-7p8rj # Cluster ID
          labels:
            field.cattle.io/projectId: p-7p8rj
        syncOptions:
        - ApplyOutOfSyncOnly=true
        - RespectIgnoreDifferences=true
        - CreateNamespace=true
        automated:
          prune: true

Inside of that repository, in branch e.g. branch_a, in folder lib/argocd/ I dont have a helm chart but rather further individual Applications -> so it is more akin to https://github.com/argoproj/argocd-example-apps/blob/master/apps approach ("App of Apps"). The individual Applications do not have managedNamespaceMetadata section on them, only syncOptions which might be slightly different for each application due to variety of reasons. The default ApplicationSet however does contain syncOptions with values. So i guess that Application has higher priority and thus it is being used for the actual deployment. And if it does not have managedNamespaceMetadata it wont inherit it as well (and that would be my expectation, to maintain it only once globally).

[crenshaw-dev]

The child applications should absolutely have the managedNamespaceMetadata field as specified in the ApplicationSet's spec.template field. I strongly suspect that either the Application or the ApplicationSet CRD is out of date on the destination cluster.

[dmpe]

The way i install argocd is via kustomization and each time everything (offico release plus my changes to it) is applied.

When I apply only crds which are 100% coming from the latest release, I dont see any changes.

$ kubectl apply -f argocd.yaml
customresourcedefinition.apiextensions.k8s.io/applications.argoproj.io unchanged
customresourcedefinition.apiextensions.k8s.io/applicationsets.argoproj.io unchanged
customresourcedefinition.apiextensions.k8s.io/appprojects.argoproj.io unchanged

What I do see is that they were updated at the end of June (in my cluster).

Status:
  Accepted Names:
    Kind:       Application
    List Kind:  ApplicationList
    Plural:     applications
    Short Names:
      app
      apps
    Singular:  application
  Conditions:
    Last Transition Time:  2023-06-28T14:01:02Z
    Message:               no conflicts found
    Reason:                NoConflicts
    Status:                True
    Type:                  NamesAccepted
    Last Transition Time:  2023-06-28T14:01:02Z
    Message:               the initial names have been accepted
    Reason:                InitialNamesAccepted
    Status:                True
    Type:                  Established
  Stored Versions:
    v1alpha1

[crenshaw-dev]

Huh. No you're absolutely right, this should be working for you. Are you 100% sure that both the applicationset-controller and application-controller are running a recent image? I'm just trying to image what could possibly be popping that field off the resource. Conceptually ApplicationSet is very simple. It just takes spec.template, applies some light transformations, and then creates or updates the Application in the cluster. Definitely shouldn't be dropping fields.

[blakepettersson]

@dmpe if it is as you say it seems more of an issue with the generator than managedNamespaceMetadata, can you provide a test case that is reproducible? Basically what you provided but with a public repository?

[dmpe]

Hmm... I believe my issue might deserve actually new issue - I will post it here for now ...

I have attempted to make a public repo with a somewhat usable example. Let's try this https://github.com/dmpe/argocd-issue-12464

1. install k8s cluster, install argocd and clone my repo (I have used minikube successfully)
2. execute manually appset.yaml in the root of the repo. kubectl apply -f appset.yaml -n argocd
3. Sync audi-apps and then both actual real life apps (did not test any GUI/api service, but all 3 should be fully "green" ...).
4. Now observe:
   • audi-apps has managedNamespaceMetadata as one would expect it to have. This is also correct, no issues here whatsoever.
   • any of those "downstream" (helm-guestbook/podinfo-example) apps are lacking managedNamespaceMetadata
   • issue: they are not inherited - should/can they be ?

There might be maybe a bit more information to share, but should I then create a new ticket first ?