[New feat] Support dependency across Helm Releases also for helm-upgrade usecases

[antaloala]

Flux2 documentation on HelmRelease dependencies clearly states it does not account for upgrade ordering, explaining the reason for that:

Note that this does not account for upgrade ordering. Kubernetes only allows applying one resource (HelmRelease in this case) at a time, so there is no way for the controller to know when a dependency HelmRelease may be updated. Also, circular dependencies between HelmRelease resources must be avoided, otherwise the interdependent HelmRelease resources will never be reconciled.

If adding a new array attribute to the .spec stanza of a HelmRelease object (e.g. naming it dependsOnVersioned) mimicking the existing dependsOn one but allowing to additionally set, per each item (i.e. per each referred HelmRelease object), the specific Helm chart version for which the dependency applies => it would then be possible for flux2 helm-controller to also support dependencies across Helm releases at (Helm) upgrade time.

Flux2 helm-controller behavior when finding .spec.dependsOnVersioned entries in a being reconciled HelmRelease object:

• read the version associated to the HelmRelease object referred as a "versioned dependency" from the being reconciled HelmRelease object
• check if the version of the Helm chart related to the latest Helm revision successfully deployed for that referred HelmRelease object matches the version indicated in the dependency (i.e. the one read in previous step)
• if it matches -> continue with the helm install/upgrade of the being reconciled HelmRelease object (if there are not more versioned dependencies to check).
• if it does not match -> reflect that in the status stanza and try it again (later).

Note this new attribute to be defined in the HelmRelease custom resource's openAPIV3Schema could be added as an optional one in order not to introduce any backward-or-forward incompatible change according to k8s API compatibility rules, so not forcing to have to define a new version of the HelmRelease custom resource in the related CRD (if not desired), so the optional attribute could be safely added to latest v2beta1 HelmRelease version of api-group helm.toolkit.fluxcd.io.

[antaloala]

@stefanprodan should I maybe move this better to the "discussions" section to have a debate on it?

[stefanprodan]

If you wrap the HelmReleases in Flux Kustomizations with wait: true then the order will be respected for upgrades. Unlike helm-controller, kustomize-controller can do this because it looks up the Git commit, then orders all Flux Kustomizations coming from the same source.

[antaloala]

Thanks @stefanprodan.

So then, in an example based on only two Helm releases "A" and "B" (where "A" depends on "B" and we want to assure the dependency applies also for Helm release upgrades) I would do the following:

1. Store both flux2 HelmRelease yamls (one per each of the two required Helm releases) in the same Git repo BUT each placed in a different path (reason for this below) (I understand we do not need to set any dependency across flux2 HelmReleases when following this approach as it is set across the flux2 Kustomization objects related to each (as proposed below), right?).

2. Create two flux2 Kustomization yamls, both referring to the same flux2 GitRepository source object BUT each pointing to one of the two paths mentioned above (i.e. each "pointing" to one of the two HelmRelease yamls). The flux2 Kustomization object's spec stanza for flux2 HelmRelease "A" object declares (using the .spec.dependsOn field) its dependency towards the flux2 Kustomization object wrapping flux2 HelmRelease "B" object This makes flux2 Kustomization "A" reconciliation to wait for flux2 Kustomization "B" object to reach the Ready condition (thanks to flux2 Kustomization custom resource's status stanza being kstatus compliant).

3. To also set wait: true in the spec stanza of flux2 Kustomization "B" object in order to assure this flux2 Kustomization object waits for the HelmRelease "B" object to reach the Ready condition and so reaching the Ready condition itself (now thanks to flux2 HelmRelease custom resource's status stanza also being kstatus compliant).

4. A single git commit/PR (applied on the source Git repo) updating both HelmRelease yamls (each pointing to newer versions of the involved Helm charts) would make the magic, assuring the (Helm) upgrade on Helm release "A" is performed once the (Helm) upgrade on Helm release "B" has been successfully run ;-)

Did I get it?

Thanks in advance ;-)

P.S: great work making all flux2 custom resources (and related custom controllers behavior) to be kstatus compliant ;-)

[antaloala]

@stefanprodan could you confirm my understanding is right .. or correct me where being wrong? (thanks a lot in advance ;-)

[stefanprodan]

@antaloala yes that sounds right to me

[antaloala]

Thanks a lot. We can then close the issue (maybe I can do (?) ... if you tell me how I am not bothering you anymore asking to close an issue I raised ;-)

Hmm ... or maybe we can use it to trigger an update in flux2 doc, explaining this possibility to cope with the limitation currently documented in HelmRelease dependencies. What do you think?

[stefanprodan]

I think it could go on the FAQ instead of helm-controller docs as Flux controllers are independent, you can use helm-controller without kustomize-controller if you chose to do so.

[antaloala]

Let me work in a doc PR (for the FAQ section) to cover this (I just forked the flux2 doc repo). Posting the PR here when having it ready. Is it ok for you to do like that?

[stefanprodan]

You need to fork the https://github.com/fluxcd/website repo, add a new FAQ entry under Helm, signoff your commit, then open a PR.

[Chili-Man]

@stefanprodan I'm having trouble understanding what's meant by wrap the HelmReleases in Flux Kustomizations , are there any examples of this you can point to?

@antaloala I'm trying to follow what you're doing, but I don't quite see the overall structure yet. Are the kustomization yamls siblings to the HelmRelease files, for example like:

.
├── a
│   ├── helm-release.yaml
│   └── kustomization.yaml
└── b
    ├── helm-release.yaml
    └── kustomization.yaml

[Chili-Man]

I've tried doing a few ways now, but for some reason, the "wrapped HelmReleases" are still getting deployed without respect to the flux kustomization declared ordering. I'm not sure what I'm doing wrong here

[souleb]

@Chili-Man I think it is better to ask on slack so we can provide examples.

[Chili-Man]

thanks @souleb , I just posted this question on Slack

[Chili-Man]

For those following along, the missing piece was that we needed to create a kustomization.yaml that basically points to the flux kustomizations so that the implicitly autogenerated kustomization.yaml by flux doesn't install the flux kustomizations in the wrong order and helm releases.

 kind: Kustomization
 resources:
   - helm-B-kustomization.yaml
   - helm-A-kustomization.yaml

[antaloala]

Sorry a lot @Chili-Man for jumping so late (I have been off for some time).

This link from flux2 documentation shows the flux2 kustomization wrapper for a flux2 HelmRelease yaml (also showing the configMapGenerator usage you can follow (I recommend to use it) once you are wrapping each flux2 HelmRelease with a flux2 kustomization.

[anshuishere]

Hello,

The current suggested solution of wrapping Flux HelmRelease with Flux Kustomizations introduces a lot of complexity in our tenant repo structure and the way I see it, the original suggestion of supporting the helm dependency on upgrades by ".spec.dependsOnVersioned" will largely reduce it. Is there something stopping us from thinking in the direction of adding "version" as part of depends on in case of Flux HelmReleases? @antaloala @stefanprodan

For example, to wrap n helm releases, we need to create n paths in the same repo and use flux kustomizations to sync them. In the final product, we may have close to a hundred helm charts, which makes this structure very complicated.

apps ├── flux_kustomization1.yaml ├── flux_kustomization2.yaml ├── flux_kustomization3.yaml ├── flux_kustomization4.yaml ├── flux_kustomization5.yaml └── helm-releases ├── helmrelease1 │ ├── helmrelease1.yaml │ └── kustomization.yaml ├── helmrelease2 │ ├── helmrelease2.yaml │ └── kustomization.yaml ├── helmrelease3 │ ├── helmrelease3.yaml │ └── kustomization.yaml ├── helmrelease4 │ ├── helmrelease4.yaml │ └── kustomization.yaml ├── helmrelease5 │ ├── helmrelease5.yaml │ └── kustomization.yaml

[jaythamke]

By the way will this strategy (of wrapping helmrelease to flux2 kustomization) also respects the deletion (reverse dependency) order that is all the dependents get deleted first and then their dependency? Thanks