Needs doesn't work properly when helmfile is distributed among multiple files

[Cayan]

I used to have a single helmfile with path parameter that defined all my helm definitions, like the structure below:

.
├── environments.yaml
├── helmfile.yaml
└── releases
    ├── namespaces.yaml
    ├── cluster-autoscaler.yaml

bases:
  - environments.yaml

helmfiles:
  - path: releases/namespaces.yaml
  - path: releases/cluster-autoscaler.yaml.gotmpl

Recently I decided to implement Renovate to update my dependencies but it only works with helmfiles, so my approach was to break my structure into multiple helmfiles.

.
├── environments.yaml
├── helmfile.yaml
└── releases
    ├── namespaces
    │   ├── helmfile.lock
    │   └── helmfile.yaml.gotmpl
    ├── cluster-autoscaler
    │   ├── helmfile.lock
    │   ├── helmfile.yaml.gotmpl

bases:
  - environments.yaml

helmfiles:
  - path: releases/namespaces/helmfile.yaml.gotmpl
  - path: releases/cluster-autoscaler/helmfile.yaml.gotmpl

Then I identified the bug where when it detects the needs group it only checks inside the same file, if one file needs something declared on a namespace on another file it no longer works.

releases/namespaces/helmfile.yaml.gotmpl

releases:
  - name: namespace
    namespace: default
    chart: zloeber/namespace
    version: 0.1.0
    values:
        ...

releases/cluster-autoscaler/helmfile.yaml.gotmpl

  - name: cluster-autoscaler
    chart: cluster-autoscaler/cluster-autoscaler
    wait: true
    namespace: kube-system
    version: 2.0.0
    needs:
      - default/namespace

Error:

in ./helmfile.yaml: in .helmfiles[6]: in releases/cluster-autoscaler/helmfile.yaml.gotmpl: release(s) "cluster-autoscaler/cluster-autoscaler/ depend(s) on an undefined release "stage/default/namespace". Perhaps you made a typo in "needs" or forgot defining a release named "namespace" with appropriate "namespace" and "kubeContext"?

[Cayan]

Seems like the --skip-needs flag doesn't work on this scenario as well

[itscaro]

IMO it's by design. Each helmfile.yaml is independant block and is executed in the order of definition in helmfiles field, as stated in README:

- # All the nested state files under `helmfiles:` is processed in the order of definition.

[rajiteh]

Ran in to the same problem, resorted to renaming the helmfiles to indicate their order of execution.

[corinz]

@itscaro Does it wait for completion of chart install? I'm having the same issue. I took the recommended "glob pattern" design by giving each chart directory a helmfile of its own. There is a master helmfile in the parent directory that include all child helmfiles with a wildcard path. I can change the directory name of the chart that is depended on to 00-mychart so it gets executed first -- but does this guarantee that it will be completely installed before others are executed?

I thought that they were executed concurrently? (This is my first day using helmfile)

Thanks for all the help

[mumoshu]

the needs group it only checks inside the same file

Yes, as @itscaro has kindly noted this is by design.

[mumoshu]

Does it wait for completion of chart install?

@corinz Yes.

but does this guarantee that it will be completely installed before others are executed?

Yes. I remember that the community decided the use of the alphanumeric ordering of the helmfile.yaml file names as the processing order would be straight-forward.

I thought that they were executed concurrently? (This is my first day using helmfile)

No, because then you have no way to order sub-helmfile runs. The concurrency is currently supported only within a single helmfile.yaml.

[hinricht]

It would be great if this limitation would be documented boldly in i.e. https://helmfile.readthedocs.io/en/latest/#dag-aware-installationdeletion-ordering-with-needs

[yxxhero]

@hinricht Please post a PR in helmfile/helmfile. THanks so much.