App of Apps not working when using kustomize applications

[ageekymonk]

Checklist:

• [ x] I've searched in the docs and FAQ for my answer: http://bit.ly/argocd-faq.
• [x] I've included steps to reproduce the bug.
• [x] I've pasted the output of argocd version.

Describe the bug

When creating an Apps of Apps, where each app is a kustomize app, apps donot get created. They throw the following error. one or more synchronization tasks are not valid. If i apply manually the app alone it works.

To Reproduce Create a kustomize app which points to another kustomize app.

Expected behavior

Apps of Apps should be created.

Screenshots

Version argocd: v1.4.2+48cced9 BuildDate: 2020-01-24T01:07:43Z GitCommit: 48cced9d925b5bc94f6aa9fa4a8a19b2a59e128a GitTreeState: clean GoVersion: go1.12.6 Compiler: gc Platform: darwin/amd64 argocd-server: v1.4.2+48cced9 BuildDate: 2020-01-24T01:07:03Z GitCommit: 48cced9d925b5bc94f6aa9fa4a8a19b2a59e128a GitTreeState: clean GoVersion: go1.12.6 Compiler: gc Platform: linux/amd64 Ksonnet Version: v0.13.1 Kustomize Version: Version: {Version:kustomize/v3.2.1 GitCommit:d89b448c745937f0cf1936162f26a5aac688f840 BuildDate:2019-09-27T00:10:52Z GoOs:linux GoArch:amd64} Helm Version: v2.15.2 Kubectl Version: v1.14.0

[shabbskagalwala]

Sorry for commenting on this issue, but I seem to be running into the same using helm app of apps too. Running a sync results in the following

argoproj.io  Application  argocd     test-app                      OutOfSync  Missing        the server could not find the requested resource

And looking at the the application object i see

status:
  health:
    status: Missing
  observedAt: "2020-05-05T00:55:28Z"
  operationState:
    finishedAt: "2020-05-05T00:55:28Z"
    message: one or more synchronization tasks are not valid
    operation:
      initiatedBy:
        username: admin

Whereas adding the applications individually via the UI seems to work without any issues.

---

argo version

➜ (⎈ gke_poc_us-central1_argo:argocd) ~  ✗ argocd version
argocd: v1.5.2+c2c19f4
  BuildDate: 2020-04-15T16:43:41Z
  GitCommit: c2c19f42ad78ed7a6fb70e86aed117be484feb50
  GitTreeState: clean
  GoVersion: go1.14
  Compiler: gc
  Platform: darwin/amd64
argocd-server: v1.5.3+095c5d6
  BuildDate: 2020-05-02T04:21:47Z
  GitCommit: 095c5d616b1cb39f87e8f7a54cabea2643e1c99a
  GitTreeState: clean
  GoVersion: go1.14.1
  Compiler: gc
  Platform: linux/amd64
  Ksonnet Version: v0.13.1
  Kustomize Version: {Version:kustomize/v3.5.4 GitCommit:3af514fa9f85430f0c1557c4a0291e62112ab026 BuildDate:2020-01-11T03:12:59Z GoOs:linux GoArch:amd64}
  Helm Version: version.BuildInfo{Version:"v3.2.0", GitCommit:"e11b7ce3b12db2941e90399e874513fbd24bcb71", GitTreeState:"clean", GoVersion:"go1.13.10"}
  Kubectl Version: v1.14.0

[Woland2k]

Same problem as above with argo 1.5.4 and kustomize templates. However, mine says sync is successful, however no application pods are created. Works fine if i create app directly. I have a full repository available if someone can double check:

https://github.com/VirtoCommerce/vc-deployments/tree/3e831b31db71fe8f9867bd6257ea69c0560c9471

After more experiements, I was able to fix the issue by adding namespace under yaml definition in the apps folder

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: webstore-demo
  **namespace: argocd**

It has to have argocd namespace, anything else and it doesn't work. In my deployment I add everything into "demo" namespace and argocd runs under default argocd namespace.

[AustinSmart]

I have the same error w/ a Helm App of Apps. All application manifests specify namespace: argocd.

edit: for anyone who found themselves here while trying out the App of App patterns. If your child apps are deploying to a different cluster, make sure your parent app has destination server: https://kubernetes.default.svc as that is where the child ArgoCD Application resources are created and tracked.

[FearTheBadger]

This is still a problem in 1.6.1

argocd version --client
  argocd: v1.6.1+159674e
  BuildDate: 2020-06-19T00:39:46Z
  GitCommit: 159674ee844a378fb98fe297006bf7b83a6e32d2
  GitTreeState: clean
  GoVersion: go1.14.1
  Compiler: gc
  Platform: linux/amd64
argocd-server version
  argocd-server: v1.6.1+159674e
  BuildDate: 2020-06-19T00:41:05Z
  GitCommit: 159674ee844a378fb98fe297006bf7b83a6e32d2
  GitTreeState: clean
  GoVersion: go1.14.1
  Compiler: gc
  Platform: linux/amd64
Kustomize version: {Version:kustomize/v3.6.1 GitCommit:c97fa946d576eb6ed559f17f2ac43b3b5a8d5dbd BuildDate:2020-05-27T20:47:35Z GoOs:linux GoArch:amd64}
Helm Version: version.BuildInfo{Version:"v3.2.0", GitCommit:"e11b7ce3b12db2941e90399e874513fbd24bcb71", GitTreeState:"clean", GoVersion:"go1.13.10"}
Kubectl Version:
  Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.0", GitCommit:"641856db18352033a0d96dbc99153fa3b27298e5", GitTreeState:"clean", BuildDate:"2019-03-25T15:53:57Z", 
  GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
  Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.0+d4cacc0", GitCommit:"d4cacc0", GitTreeState:"clean", BuildDate:"2019-10-31T21:06:55Z", GoVersion:"go1.10.8", Compiler:"gc", Platform:"linux/amd64"}

The error from the application controller:

level=info msg="adding resource result, status: 'SyncFailed', phase: '', message: 'the server could not find the requested resource'" application=test kind=Kustomization name= namespace=test phase=Sync syncId=00008-PFwkm
level=info msg="Updating operation state. phase: Running -> Failed, message: '' -> 'one or more synchronization tasks are not valid'" application=test syncId=00009-cRIAm
level=info msg="sync/terminate complete" application=test duration=7.003573ms syncId=00009-cRIAm
level=info msg="updated 'test' operation (phase: Failed)"
level=info msg="Sync operation to 056d4f65824e28726b07ca417ed9452cdbd30db5 failed: one or more synchronization tasks are not valid" application=test dest-namespace=test dest-server="https://kubernetes.default.svc" reason=OperationCompleted type=Warning

[flickerfly]

One thing I've discovered in my research is the API endpoint yourargocd.com/api/v1/repositories/{repo}/apps. It appears that if if hit, with the problematic {repo} name, it would result in some context for where the issue manifests. It is document on your ArgoCD UI at the path yourargocd.com/swagger-ui#operation/ListApps.

I'm wondering, is it that ArgoCD isn't recognizing Kustomize is involved so trying to actually add the Kustomization type to the cluster rather than processing it? If so, how does ArgoCD get triggered on this?

In the interest of discovering that information, but not being a Golang guy, I found these hopefully relevant bits. There is apparently two ways this is supposed to be triggered. There is some documentation in the Tool Detection Doc, but it left me with some questions and clearly it isn't working exactly as documented.

First, auto-detection based on the contents of the repo.

So what does the code use to determine if a directory of code should be handled as a Kustomize appType? The answer appears to be on line 168 of kustomize.go.

var KustomizationNames = []string{"kustomization.yaml", "kustomization.yml", "Kustomization"}

This is then used on line 181 in the isKustomization function. It appears very consistent with the docs.

This info is used by reposerver/repository/repository.go on line 100 in the ListApps function. There is also a ListApps function in server/repository/repository.go on line 155, but this one doesn't use the discovery.Discover function, but instead calling the API ListApps endpoint.

Second, explicit identification in the Application resource as spec.source.kustomize. I'm not clear on what this should be if I just want to explicitly say this should use kustomize, but not change other characteristics. Maybe spec.source.kustomize: []? pkg/apis/application/v1alpha1/types.go line 2260 seems to be involved and best I can tell anything non-nil would be acceptable.

The two methods of detection appear to come together in reposerver/repository/repository.go function GetAppSourceType where it first checks ExplicitType, then if that comes back blank, uses discovery.AppType to tell GenerateManifests which type of application to output manifests for and hopefully hits line 378 where it gets processed by Kustomize.

My suspicion is that we aren't making it this far, but I don't know how to watch the code at work.

[FearTheBadger]

I think I have finally found the trigger point.

All kustomization.yaml files will be ignored, or kind: Kustomization configs will throw the error above, if the application source directory recurse: true is set. recurse: false or omitting recurse works without issue.

Failing scenario:

kind: Application
spec:
  source:
    directory:
      recurse: true

Successful scenario:

kind: Application
spec:
  source:
    directory:
      recurse: false

[dpennell]

edit: for anyone who found themselves here while trying out the App of App patterns. If your child apps are deploying to a different cluster, make sure your parent app has destination server: https://kubernetes.default.svc as that is where the child ArgoCD Application resources are created and tracked.

This works for app of apps where all of the children are apps. If any of the children are simple resources (e.g. Secret), the end up being created in the local cluster and not the external one.

[bitva77]

is this really still an issue? I'm POC'ing ArgoCD now but this bug makes it seem like App of Apps configuration is broken and so what's the point

[bobbywatson3]

I'm in the same boat as @bitva77 . Our app of apps is a mixture of applications (off the shelf helm charts) and template files in the app of apps chart. Is that not supported?

[flickerfly]

It does seem to work, but there are some gotchas along the way. The one that had us really twisted is that you can NOT set up the app to be recursive or it doesn't properly detect what is going on and ArgoCD gets terribly confused.

[bobbywatson3]

I think I have finally found the trigger point.

All kustomization.yaml files will be ignored, or kind: Kustomization configs will throw the error above, if the application source directory recurse: true is set. recurse: false or omitting recurse works without issue.

Failing scenario:

kind: Application
spec:
  source:
    directory:
      recurse: true

Successful scenario:

kind: Application
spec:
  source:
    directory:
      recurse: false

I tried this, and the app stayed out of sync. It could never sync the directory recurse section. Apparently recurse: false is the default behavior.

I ended up not using this tiny kustomize file, instead copying in the entire 26,000 line cert-manager kube file. Bummer, but it works.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: cert-manager
resources:
- github.com/jetstack/cert-manager/releases/download/v1.1.0/cert-manager.yaml

[riweston]

I think I have finally found the trigger point.

All kustomization.yaml files will be ignored, or kind: Kustomization configs will throw the error above, if the application source directory recurse: true is set. recurse: false or omitting recurse works without issue.

Failing scenario:

kind: Application
spec:
  source:
    directory:
      recurse: true

Successful scenario:

kind: Application
spec:
  source:
    directory:
      recurse: false

Is there any way around this? I'm specifically trying to use a number of kustomize.yml configs in my repo and rely on all applications being recursively deployed from a kustomize file but I'm seeing the same sync issues unless I disable recursion

In my case I'm trying to run a bootstrap that then adds an app-of-apps to argocd where I expect my source of truth for everything to be.

.
├── argocd
│  ├── argocd
│  │  └── kustomization.yml
│  └── sealed-secrets
│     └── kustomization.yml
├── bootstrap
│  └── kustomization.yml
├── juju
│  ├── deployment.yml
│  └── README.md
├── kustomize
│  ├── argocd
│  │  ├── app.yml
│  │  ├── cm.rbac.yml
│  │  ├── cm.repos.yml
│  │  ├── cm.sso.yml
│  │  ├── kustomization.yml
│  │  ├── overlay.argocd.yml
│  │  ├── overlay.secrets.yml
│  │  ├── resource.ingress.yml
│  │  ├── resource.namespace.yml
│  │  ├── resource.project.yml
│  │  └── resource.secrets.yml
│  └── sealed-secrets
│     ├── app.yml
│     └── kustomization.yml
└── README.md

# argocd/argocd/kustomization.yml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - ../../kustomize/argocd/

# kustomize/argocd/app.yml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: argocd
  namespace: argocd
spec:
  source:
    path: argocd
    directory:
      recurse: true
...

[bygui86]

Hi guys, any news about this issue?

[selcuk-cofe]

That issue still exist I m using 2.0.5 and in the middle of the POC with kustomization app of apps repo. I already tried @Woland2k and @FearTheBadger recommendation but still have no luck child apps stuck at OutOfSync

argocd: v2.0.5+4c94d88.dirty BuildDate: 2021-07-23T05:12:02Z GitCommit: 4c94d886f56bcb2f9d5b3251fdc049c2d1354b88 GitTreeState: dirty GoVersion: go1.16.6 Compiler: gc Platform: darwin/amd64

[trajakovic]

I can confirm. This issue still exists to this date: { "Version": "v2.1.6+a346cf9", "BuildDate": "2021-10-28T19:59:40Z", "GitCommit": "a346cf933e10d872eae26bff8e58c5e7ac40db25", "GitTreeState": "clean", "GoVersion": "go1.16.5", "Compiler": "gc", "Platform": "linux/amd64", "KsonnetVersion": "v0.13.1", "KustomizeVersion": "v4.2.0 2021-06-30T22:49:26Z", "HelmVersion": "v3.6.0+g7f2df64", "KubectlVersion": "v0.21.0", "JsonnetVersion": "v0.17.0" }

What it does, ArgoCD recognised generated resource (in my case Deployment), but ended with "Sync Failed" and message: "the server could not find the requested resource"

[jordan-seemplicity]

I suppose it makes sense that this wouldn't work, since there is ambiguity in how to handle a directory containing both an Application manifest as well as a Kustomize entrypoint in some sub-directory with recursion enabled - do we deploy the Kustomize manifest as part of the base app or as part of the dependent app?

It would be nice if Argo was "smarter" about how to deploy the app of apps when using Kustomize, i.e., allow recursive directories, but only for Application manifests. Maybe implementing some kind of globbing filter would help:

spec:
  directory:
    recurse: true
    pattern: **/app.yaml

For the time being, my seed app is set to directory.recurse: false and pointing to a flat directory of app manifests (clusters/cluster-1). Each app manifest points to its own sub-directory (clusters/cluster-1/my-app), containing Kustomize manifests or Helm charts. Recursion can be independently configured for each app as needed.

[marcofranssen]

I'm facing the same issue and can't figure out why it isn't working. I'm sure it has worked at some point.

See here my app of apps.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: opa-policies
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  destination:
    namespace: opa-policies
    server: {{ .Values.spec.destination.server }}
  project: my-project
  source:
    path: opa-policies
    repoURL: {{ .Values.spec.source.repoURL }}
    targetRevision: {{ .Values.spec.source.targetRevision }}
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true

Then in the folder opa-policies I have the following Kustomize.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

# namespace: gatekeeper-system

resources:
  # - https://github.com/sigstore/cosign-gatekeeper-provider/raw/main/manifest/deployment.yaml
  # - https://github.com/sigstore/cosign-gatekeeper-provider/raw/main/manifest/service.yaml
  # - https://github.com/sigstore/cosign-gatekeeper-provider/raw/main/manifest/provider.yaml
  - templates/container-resources-template.yaml
  - templates/image-tagging-template.yaml
  - templates/cosign-template.yaml
  - constraints/image-must-not-have-latest-tag-constraint.yaml
  - constraints/image-must-have-limits-and-requests-constraint.yaml
  - constraints/image-signed-constraint.yaml

# patches:
#   - target:
#       kind: Provider
#       group: externaldata.gatekeeper.sh
#       version: v1alpha1
#       name: cosign-gatekeeper-provider
#     patch: |-
#       - op: add
#         path: /metadata/annotations/argocd.argoproj.io~1sync-wave
#         value: -2
#       - op: add
#         path: /spec/insecureTLSSkipVerify
#         value: true
#       - op: replace
#         path: /spec/timeout
#         value: 15
#       - op: replace
#         path: /spec/url
#         value: http://cosign-gatekeeper-provider.gatekeeper-system:8090/validate
#   - target:
#       kind: Deployment
#       group: apps
#       version: v1
#       name: cosign-gatekeeper-provider
#     patch: |-
#       - op: add
#         path: /metadata/annotations/argocd.argoproj.io~1sync-wave
#         value: -2
#   - target:
#       kind: Service
#       version: v1
#       name: cosign-gatekeeper-provider
#     patch: |-
#       - op: add
#         path: /metadata/annotations/argocd.argoproj.io~1sync-wave
#         value: -2

The commented lines I have tested with to figure out if it is related to any kustomize specifics in combination with ArgoCD.

The result is that ArgoCD thinks all is synchronized, however the App is not created.

time="2022-08-16T09:13:35Z" level=info msg="Applying resource Config/config in cluster: https://172.20.0.1:443, namespace: gatekeeper-system"
time="2022-08-16T09:13:35Z" level=info msg="Adding resource result, status: 'Synced', phase: 'Running', message: 'config.config.gatekeeper.sh/config unchanged'" application=my-policies kind=Config name=config namespace=gatekeeper-system phase=Sync syncId=00052-MtTBa
time="2022-08-16T09:13:35Z" level=info msg="Applying resource Application/opa-policies in cluster: https://172.20.0.1:443, namespace: argocd"
time="2022-08-16T09:13:35Z" level=info msg="Adding resource result, status: 'Synced', phase: 'Running', message: 'application.argoproj.io/opa-policies unchanged'" application=my-policies kind=Application name=opa-policies namespace=argocd phase=Sync syncId=00052-MtTBa

See this screenshot all is synced according to ArgoCD, but when clicking the red circled icon the app doesn't actually exist, neither it is applied.

See here the screenshot showing the app that pretends to by synced actually isn't.

Strange thing is there are no errors, and nothing in the logs.

[marcofranssen]

Thanks to @crenshaw-dev I was able to resolve the issue.

Solution

All applications have to be installed in the argocd namespace.

So like

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: policies
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
  …

Following is not yet supported and will probably land in the next 2.5.x release.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: policies
  namespace: my-other-namespace
  finalizers:
    - resources-finalizer.argocd.argoproj.io
  …

So namespace: argocd is a hard requirement with current release of ArgoCD.

[winston0410]

I think I have finally found the trigger point. All kustomization.yaml files will be ignored, or kind: Kustomization configs will throw the error above, if the application source directory recurse: true is set. recurse: false or omitting recurse works without issue. Failing scenario:

kind: Application
spec:
  source:
    directory:
      recurse: true

Successful scenario:

kind: Application
spec:
  source:
    directory:
      recurse: false

I tried this, and the app stayed out of sync. It could never sync the directory recurse section. Apparently recurse: false is the default behavior.

I ended up not using this tiny kustomize file, instead copying in the entire 26,000 line cert-manager kube file. Bummer, but it works.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: cert-manager
resources:
- github.com/jetstack/cert-manager/releases/download/v1.1.0/cert-manager.yaml

I am getting an error mentioned by this ticket, when I tried to use a yaml with kind: Kustomization. How did you workaround and apply that yaml with argocd at the end?

The error that I got:

2022-09-26T23:36:30+01:00  kustomize.config.k8s.io  Kustomization     default                        OutOfSync  Missing              the server could not find the requested resource

[Xtigyro]

+1

[rafael-adorna-incode]

+1

[juangb]

+1

[sjonpaulbrown-cc]

This should be supported since v0.2.5. The following documentation outlines how to configure argo to deploy to any namespace. After initially configuring argo, I have configured it to self-manage itself, and using the below configuration, I have updated it to support deployments to any namespace.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: argocd
  namespace: argo
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: cd
  sources:
    - repoURL: https://github.com/argoproj/argo-cd.git
      targetRevision: v2.10.6
      path: examples/k8s-rbac/argocd-server-applications

    - chart: argo-cd
      repoURL: https://argoproj.github.io/argo-helm
      targetRevision: 6.7.11
      helm:
        valuesObject:
          configs:
            application.namespaces: "*"
          redis-ha:
            enabled: false
          controller:
            replicas: 1
          server:
            autoscaling:
              enabled: true
              minReplicas: 2
          repoServer:
            autoscaling:
              enabled: true
              minReplicas: 2
          applicationSet:
            replicas: 2
  destination:
    server: "https://kubernetes.default.svc"
    namespace: argo
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

The key points here are that the configs section of the values needs to have application.namespaces: "*" set to your namespaces or any namespace. It would likely be best to isolate these to only the namespaces that you want to control with argo rather than leaving it open with a wildcard.

          configs:
            application.namespaces: "*"

Along with that, the additional RBAC resources must be created to grant argo the access it needs. This is handled in the other source that I had outlined.

    - repoURL: https://github.com/argoproj/argo-cd.git
      targetRevision: v2.10.6
      path: examples/k8s-rbac/argocd-server-applications

After updating the config & RBAC, you can now create argo Applications in new namespaces. When creating the Application, you need to make sure that your project supports the source namespace. This can be added to your project using the following. Once again, it would be best practice to limit the namespaces to only those that you want to support rather than leaving it wide open. To be fully clear, the sourceNamespace is specific to the Argo Applications that are deploying resources to a specific destination namespace.

spec:
  sourceNamespaces:
    - "*"

After adding this, I can deploy the argo application to a separate namespace, and I have been able to successfully deploy a combination of helm resources & manifests to the the same destination namespace from a single app of apps.

[crenshaw-dev]

Closing because I think most of the issues here are due to a misunderstanding of how an app-of-apps works.

An app-of-apps isn't really anything special. It's just an Application CR that deploys the manifests that happen to be other Applications.

Applications must live on the same cluster as the Argo CD app controller, either in the same namespace as the app controller, or in an explicitly-configured additional namespace using the "apps in any namespace" feature.

In other words, the spec.destination of the parent app is always the cluster Argo CD is running on.

The resources managed by the child Applications could theoretically be any kind and could be destined for any namespace on any cluster.

If you have a directory structure like this:

apps/
  app-1/
    application.yaml
    resources/
      configmap.yaml
      deployment.yaml
  app-2/
    ...

then you need to make sure your app-of-apps' child Applications have recursion turned off. Otherwise the parent app is going to pick up configmap.yaml and deployment.yaml and try to deploy them to the Argo CD cluster in the app controller's namespace (at least, if your parent app is configured correctly).

If you have a directory structure like this:

apps/
  app-1/
    application.yaml
    configmap.yaml
    deployment.yaml
  app-2/
    ...

then disabling recursion is not enough -- you need to configure the parent app to ignore everything besides the application.yaml files.

Important: I don't recommend keeping your Application manifests in the same repo as other resources, unless the people writing/reviewing/merging those resources are Argo CD admins or anyway or unless you're using the Apps-in-any-namespace pattern with a carefully-crafted parent app. If a random user can set the spec.project field in an Application, they can use any project they want, even one with more permissions than they should have.

Hope that's clear. Closing since I think this is a collection of misunderstandings of how Apps-of-apps work. Feel free to continue to discuss though!