helmfile template cleanup output directory

[shibumi]

Hi,

I use helmfile template for generating resources for ArgoCD and then I apply these generated resources. Right now I have to do rm -rf k8s && helmfile template --output-dir k8s when I generate the resources.

Would an additional flag for deleting the output dir first make sense? Or is this something that should be scripted anyway?

Why doing rm -rf at all: The problem is that helmfile template only "adds" additional files. If I delete resources from the helmfile the old resources will still exist and this is a problem, because this way they will never get removed.

[mumoshu]

@shibumi Hey! Interesting. Running rm -rf makes sense. I'm rather willing to enhance helmfile around it but not sure how. What would be an ideal flag name for that? helmfile template --output-dir k8s --sync, helmfile template --output-dir k8s --rm-dir, helmfile template --output-dir k8s --empty-before-run are what I came up with. But none of these sounds good to me.

[shibumi]

@mumoshu thanks for your answer :) good question. I have no idea. I think something like --force-empty would be clear. I also don't know if just deleteting the whole directory is the nicest way to do this. It's definitely the easiest way, but also the most dangerous. Just imagine you mistyped the output-dir and you delete a whole folder that shouldn't be deleted. If we decide for rm -rf or the Go equivalent for it, we should do a few checks before:

1. path must be relative. A path with leading "/" shouldn't be allowed because somebody could delete their "/". Alternative: just check for "/" as string.. :thinking:
2. We could drop a ".helmfile-created" file in the top of the directory, just empty.. that way we know if it's a helmfile generated file or not and it's clear that helmfile is allowed to delete it.
3. We could really "sync" instead of rm -rf the whole directory. Maybe with dumping everything to /tmp and then doing an rsync like operation: https://github.com/Redundancy/go-sync

[mumoshu]

@shibumi Thanks! Well, we'd better create another command dedicated to output gitops config for ArgoCD. It should work mostly like helmfile template, but somehow detects "already installed releases" from the state of the target directory and treat any missing directory in the in-memory desired state as uninstalled release(s).

do you have any specific requirement on where each release should be output? Or are you literally using --output-dir k8s only and not --output-dir-template and you don't need anything special regarding it?

[shibumi]

@mumoshu Right now I just do helmfile template --output-dir k8s. Then I just use k8s as path for argoCD. My only problem right now is that some resources are missing, but for this I've found forceNamespace already (that I still need to test).

The directory structure is fine for me, I just don't know if the first layer is necessary. I think a structure like:

k8s -> cluster context(?) -> namespace -> name of release

would be sufficient.

Right now it looks like this:

k8s
├── helmfile-08b89c48-mariadb-10-3
│  └── mariadb
│     └── templates
│        ├── master-configmap.yaml
│        ├── master-statefulset.yaml
│        ├── master-svc.yaml
│        ├── slave-configmap.yaml
│        ├── slave-statefulset.yaml
│        └── slave-svc.yaml

This feature could be especially interesting if we have the new release with sticking environments to cluster contexts :)

The best solution would be helmfile support in ArgoCD, but the devs of Argo seem to not be convinced yet.

Btw: did you know that Jenkins X new version 3.0 uses helmfile?

[mumoshu]

@shibumi Thanks! Makes sense.

The directory structure is fine for me, I just don't know if the first layer is necessary.

I thought the first layer was to help NOT overwriting previously written directory and I thought I've used the hash (08b89c48 in your example) computed from the rendered helmfile.yaml content for that. But as you're worrying about subsequent helmfile-template runs overriding the dir, perhaps the hash value and the first layer isn't changing when necessary?

I think you'll probably be interested in a few issues @katsew has written recently and the underlying use-case we can see from his explanations.

https://github.com/roboll/helmfile/issues?q=is%3Aissue+is%3Aclosed+author%3Akatsew

In #1782, he mentions --output-dir-template '{{ .OutputDir }}/{{ .Release.KubeContext }}/{{ .Release.Namespace }}/{{ .Release.Name }}' which looks exactly like what you want.

Btw: did you know that Jenkins X new version 3.0 uses helmfile?

I didn't know exactly but I was wondering, as @jstrachan and his colleague occasionally come and submit PRs to Helmfile 😃

[shibumi]

I thought the first layer was to help NOT overwriting previously written directory and I thought I've used the hash (08b89c48 in your example) computed from the rendered helmfile.yaml content for that. But as you're worrying about subsequent helmfile-template runs overriding the dir, perhaps the hash value and the first layer isn't changing when necessary?

Yes, but in case of gitops we want to overwrite it. The changes are in the git directory anyway :D So it's fine to override the output-dir.

Thanks for pointing me to the output-dir-template command. This looks like it's exactly what I want and maybe I can just script around it with rsync or rm -r output-dir.

[mumoshu]

Yes, but in case of gitops we want to overwrite it. The changes are in the git directory anyway :D So it's fine to override the output-dir.

@shibumi Alright then :) I was just wondering if it's really working as my intention. Anyway, --output-dir-template should better fit your use-case and you shouldn't care much about the hash value now.

Would you mind sharing me your solution and the script, if you managed to make it work? That would help designing how our helmfile-template-like command dedicated for ArgoCD integration should look like.

[shibumi]

@mumoshu sounds good.

[katsew]

Let me share a tiny tips for this case.
Using prepare hooks to rm -rf output-dir , combined with --output-dir-template.

e.g.

releases:
- name: mariadb
  hooks:
  - events: ["prepare"]
    showlogs: true
    command: "bash"
    args:
    - -c
    - rm -rf ./k8s/{{`{{.Release.KubeContext}}`}}/{{`{{.Release.Namespace}}`}}

helmfile template --output-dir ./k8s --output-dir-template "{{ .OutputDir }}/{{ .Release.KubeContext }}/{{ .Release.Namespace }}"

This could work but be aware that the hook is executed with relative path from the helmfile.yaml which user executes. I think this is kind of "helmfile way" to do things before templating.

Hope it's useful for you.

[croissong]

Very useful, but unfortunately this also removes the output directory when running helmfile list :/