ApplicationSet not allowing environment-specific values.yaml files outside the chart directory

[tomlnd]

Checklist:

• [x] I've searched in the docs and FAQ for my answer: https://bit.ly/argocd-faq.
• [x] I've included steps to reproduce the bug.
• [x] I've pasted the output of argocd version.

Describe the bug

I'm trying to create an ApplicationSet that would deploy 2 apps (a frontend & a backend in separated charts) from inside apps/ folder in git, and choose the values.yaml for them from tenants/ folder, which is in the git repo's root instead of under the charts' folder.

This results in this error:

Failed to load target state:
failed to generate manifest for source 1 of 1: rpc error: code = Unknown desc = Manifest generation error (cached):
`helm template . --name-template frontend-application --namespace env-xxx --kube-version 1.27 --values <path to cached source>/tenants/*.yaml <api versions removed> --include-crds` failed
exit status 1: Error: open <path to cached source>/tenants/*.yaml: no such file or directory

The repo structure would look like this:

.
├── apps
│   ├── frontend
│   │   ├── Chart.lock
│   │   ├── charts
│   │   ├── Chart.yaml
│   │   ├── templates
│   │   └── values.yaml
│   └── backend
│       ├── Chart.lock
│       ├── charts
│       ├── Chart.yaml
│       ├── templates
│       └── values.yaml
└── tenants
    ├── dev-values.yaml
    └── prod-values.yaml

And the ApplicationSet like this:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: example-appset
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - git:
        repoURL: 'https://gitlab.com/example/gitops-repo.git'
        revision: master
        directories:
          - path: apps/frontend
          - path: apps/backend
  template:
    metadata:
      name: '{{ .path.basename }}-application'
    spec:
      project: default
      sources:
        - repoURL: 'https://gitlab.com/example/gitops-repo.git'
          targetRevision: master
          path: '{{ .path.path }}'
          helm:
            valueFiles:
              - /tenants/*.yaml
      destination:
        server: https://kubernetes.default.svc
        namespace: 'env-{{ .path.basename }}'
      syncPolicy:
        automated: {}
        syncOptions:
          - CreateNamespace=true

Is it prohibited by design to access values.yaml files outside the chart folder itself? This way we could add new envs / tenants just by adding a new values.yaml file under the /tenants/ folder with edited parameters, compared to the charts' default values.yaml in apps/frontend/values.yaml and apps/backend/values.yaml. Or would there be a better way of managing these, that would also work with ArgoCD?

Version

v2.12.6

[aali309]

I will look into this.

[aali309]

@andrii-korotkov-verkada can you please assign me this issue as well. Thnx.

[carlosrodfern]

@tomlnd, yes, by design, the values must be inside the chart; including your environment specific ones. If you really want them separate, then you would need to use multi-source, and the helm chart should be published in some sort of internal helm repo, and your environment-specific values would go into your git repo.

[tomlnd]

I got this working the way I originally wanted (keeping everything in one repo, and have tenant-specific values.yaml files outside the charts' folders) by playing around with the generators.

I switched the generators[].git.directories[].path value to point to the tenants' folder at repo root, instead to the /apps folder, like previously. And added a list-generator for the apps I wanted to include in this applicationset under the /apps folder.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: example-appset
  namespace: argocd
spec:
  generators:
    - matrix:
        generators:
          - git:
              repoURL: 'https://gitlab.com/example/gitops-repo.git'
              revision: master
              directories:
                - path: 'tenants/*'
          - list:
              elements:
                - app: backend
                - app: frontend
  template:
    metadata:
      name: 'tenant-{{path.basename}}-{{app}}'
    spec:
      project: production
      source:
        repoURL: 'https://gitlab.com/example/gitops-repo.git'
        targetRevision: master
        path: 'apps/{{app}}'
        helm:
          valueFiles:
            - '/tenants/{{path.basename}}/values.yaml'
      destination:
        server: 'https://kubernetes.default.svc'
        namespace: 'tenant-{{path.basename}}'
      syncPolicy:
        syncOptions:
          - CreateNamespace=true
        automated:
          prune: true
          selfHeal: true

The repo structure is now the following:

.
├── applicationset.yaml
├── apps
│   ├── backend
│   │   ├── Chart.yaml
│   │   └── values.yaml
│   └── frontend
│       ├── Chart.yaml
│       └── values.yaml
└── tenants
    ├── alice
    │   └── values.yaml
    └── bob
        └── values.yaml

This successfully creates 2 namespaces (tenant-alice & tenant-bob), and creates 2 applications per tenant (tenant-alice-frontend, tenant-alice-backend & tenant-bob-frontend, tenant-bob-backend), while keeping everything in one repo and having the tenants' values.yamls separated.

NAMESPACE   NAME                          SYNC STATUS   HEALTH STATUS
argocd      tenant-alice-backend          Synced        Healthy
argocd      tenant-alice-frontend         Synced        Healthy
argocd      tenant-bob-backend            Synced        Healthy
argocd      tenant-bob-frontend           Synced        Healthy

[aali309]

@carlosrodfern, if this is by design can we close this issue with Working as Designed comment? And may be document the matrix generator solution as the recommended approach?

[carlosrodfern]

@carlosrodfern, if this is by design can we close this issue with Working as Designed comment? And may be document the matrix generator solution as the recommended approach?

The OP @tomlnd was able to find an awesome work around. Yes, it is working by design as far as I know, and the solution is rather just working with the tool to make it happen. I'll let the OP comment whether it is good to close this ticket...

[tomlnd]

I'm just wondering why is it prohibited to let ArgoCD access the overriding values.yaml files outside the chart's directory, if I can still do that via my workaround that is not documented anywhere (should it be)? Seems a bit counterproductive in my opinion. Has there been a discussion about this design choice before in another issue?

Keeping tenants' configuration files under their own directories makes managing them easy in a multi-tenant environment., instead of having them all under /apps/frontend/tenants/... directory for example, and the same for the backend chart, and so on.

[carlosrodfern]

For context: https://argo-cd.readthedocs.io/en/stable/user-guide/helm/#values-files

Before v2.6 of Argo CD, Values files must be in the same git repository as the Helm chart. The files can be in a different location in which case it can be accessed using a relative path relative to the root directory of the Helm chart. As of v2.6, values files can be sourced from a separate repository than the Helm chart by taking advantage of multiple sources for Applications.

The problem with multi-source approach is that you can't rollback as you would do with helm rollback.

[tomlnd]

Right, that clears it. This can be closed now.