Add support for ArgoCD resource hooks and sync waves

[mumoshu]

I've often heard that people use helmfile template to export the whole collection of K8s manifests managed by Helmfile to a "config repo" for GitOps, so that actual deployments can be delegated to ArgoCD and Flux.

The advantages of this approach are:

• Higher security: Your CI doesn't need production env access for diffing and applying anymore
• Better visibility: Instead of helmfile diff results being posted in your Slack channel or pull request comments, they are now visible and reviewable in the pull request's "Files changed" tab would give you better reviewing experience by providing nicer coloring, layouts, inline review comments, etc.

Altohugh Helmfile + GitOps do have advantages over the "Helmfile on CI" approach, there are issues, too.

For me, the biggest issue in traditional GitOps had been that there was no way to control installation/upgrade order of helm charts. Helmfile have been long supporting this via DAG(needs: ...) and sub-helmfiles(helmfiles: [...]).

However, I've recently got to know about ArgoCD's Resource Hooks and Sync Waves, and realized that, theoretically, it enables Helmfile to export manifests with ordering in mind.

I'd like to experiment this by introducing a new flag --argocd-phases-and-hooks to helmfile template.

The flag will instruct Helmfile to add argocd.argoproj.io/hook: PreSync annotations to whatever manifest that has "helm.sh/hook": pre-install,pre-upgrade, and also argocd.argoproj.io/sync-wave: "N" where N is the globally assigned sequence number of the set of releases that are installed/upgraded concurrently.

To illustrate how N is assigned, let's assume you have the following helmfiles:

helmfile.yaml:

helmfiles:
- infra.yaml
- apps.yaml

apps.yaml:

releases:
- name: foo-api
  chart: mycharts/api
  # ...
- name: bar-api
  chart: mycharts/api
  # ...
- name: web
  chart: mycharts/web
  needs:["foo-api", "bar-api"]

infra.yaml

- name: logging
  chart: mycharts/fluentbit
  # ...
- name: prometheus
  chart: mycharts/prometheus
  # ...
- name: cilium
  chart: cilium/cilium
  # ...
- name: istio
  chart: istio/istio
  # ...
  needs: ["cilium"]

Today, helmfile sync and helmfile apply deploys this in the following order. Note that each group of releases are deployed concurrently as they don't depend on each other.

1: logging, prometheus, cilium

2. istio
3. foo-api, bar-api
4. web

helmfiel template --argocd-phases-and-hooks will give argocd.argoproj.io/sync-wave: "1" annotations to any manifest that belongs to logging, prometheus and cilium, argocd.argoproj.io/sync-wave: "2" for istio, and so on.

You can git-push these manifests to a config repo, so that ArgoCD will sync those manifests in that order.

[travisghansen]

Interesting thought this. I see some issues as well (for example what if the chart is already using the annotations for items within the chart).

On a related note, I built a simple helmfile integration with argo-cd that may be of interest to you @mumoshu https://github.com/travisghansen/argo-cd-helmfile

[mumoshu]

@travisghansen Awesome! Thanks for sharing your work.

As you've shown in your examlpe, helmfile template can be used in two ways - within ArgoCD to render manifests immediately before sync, or within your CI to render manifests to be git-committed into the config repo(so that ArgoCD just sees raw k8s manifests, w/o helmfile used as a config management plugin).

I suppose this proposal would add ordering to k8s resources for both ways and nothing else.

for example what if the chart is already using the annotations for items within the chart

You're referring to cases like k8s resources rendered from your chart is already containing argocd-specific annotations?

[travisghansen]

@mumoshu yeah, charts that already use the argo specific annotations correct.

Understood on the 2 approaches, my plugin is attempting the former of the 2 options. You can store your configs in git and/or put a full helmfile.yaml as an environment var for the app as well.

[msutter]

@mumoshu I'm running a continuous deployment workflow with following tools.

• gopass (for secrets)
• summon (to inject secrets as env vars or complete values.yaml files)
• helmfile (release definitions only template)
• argocd for deployment

As we want to switch from helm to argocd for the final deployment, the goal is to continue to use helmfile for manifest generation and kustomize transformers.

So far, so good.

We also have helm test Pod's (Provided by developer teams and with helm.sh/hook annotations) which I try to convert to argocd postSync annotations (argocd.argoproj.io/hook: PostSync). The newly JsonPatch (helm-x) integration you just released yesterday is just perfect for my use case. (thanks for that by the way)

With latest version of helmfile (0.118.1), I'm facing following issue (bug ?)

'helmfile template' output works fine, outputting my helm test pod manifests. But if I add the '--output-dir' option, helmfile skips the pod manifest if there is an 'helm.sh/hook' annotations in it. Therefore I'm not able to use it or patch it with kustomize.

For clarification, in the paste I did a bash script to create the kustomization bases for each release created by 'helmfile template --output-dir' and used it in a final kustomize process. Now with your helm-x integration I only need to use it for kustomize transformers not yet supported by helmfile.

Back to my issue. The same skipping seems to appear when using integrated helm-x kustomize transformers (as I guess you use the output-dir behind the scene).

If I remove the annotation 'helm.sh/hook', everything works nice. But I cannot remove it on upstream helm charts, and I think helm test is a really good way to provide integration tests for a dedicated application. (We use tests based on the robot testing framework).

Sorry, long story and not sure if it's the right place for my issue. But as this ticket describes ArgoCD resource hooks and sync waves, would be great to consider also helm test pods.

Should I open an other issue for the 'helm.sh/hook' annotations ? eg. "helmfile template --output-dir skips objects with 'helm.sh/hook' annotations"

cheers

[dudicoco]

I've read about both ArgoCD and Flux and I don't see any advantage in using either of these systems along with helmfile.

Our helmfile CI/CD process is very simple:

1. Get diffed files from commit
2. Run a check as part of the CI: helmfile apply --args "--dry-run" with an added selector for each changed release.
3. Review both the diffed files in git and the output of helmfile which is delivered as a comment.
4. After merge, run the same helmfile apply command but without the --dry-run flag

[dudicoco]

Ok so I now see one huge value in using Argo CD - the UI! Currently helm does not provide any progress information about your on going deployment which is a big problem - you need to manually check the state of the deployment resource, the pods, the logs etc.

@mumoshu, I believe the CI system should diff against the actual state of the cluster, just like terraform plan is checking the actual resources to show a real diff. Currently helm-diff does not provide this - https://github.com/databus23/helm-diff/issues/176. I agree that the dependency graph from helmfile is a must when using Argo CD. Wouldn't the ideal solution be to open a PR in the Argo CD project to add this feature?

The main problem I see with Argo CD is this - how do you deploy the Argo CD components? Helm/Helmfile are only client side so everything is very easy, Argo CD is server side, so it must be first installed by a different tool, but it might have dependencies itself in order for it to be installed such as an ingress controller, external-dns and a secrets controller. So this is a serious chicken and egg problem.

[mumoshu]

@dudicoco I think you'd usually install and manage Argo CD itself with helmfile apply, while managing your apps with ArgoCD Application custom resources pointed to gitops config repo(containing K8s manifests) generated by helmfile template

This way, you can also diff against (mostly) the actual cluster state with git-diff or github or diff, as ArgoCD guarantees the manifests committed to the config repo are periodically applied to the live cluster.

[dudicoco]

Looks like proper dependencies between applications will be implemented into Argo CD: https://github.com/argoproj/argo-cd/pull/3892

[mumoshu]

@dudicoco Thanks for sharing! The feature does seem like a better fit when you're to create Application custom resource for each sub-directory generated by helmfile template --output-dir.

When you need ordering within a single subdirectory(release), apparently we still need sync waves/resource hooks.

[abdennour]

i like it.