Deleting Helm releases

[TBeijen]

Kube-janitor seems perfect for auto-cleaning of temp. feature deploys. But... we use Helm. So it would remove the resources and leave the release data (secret objects) dangling.

Possible ways to cleanup helm releases could be:

Determine release and use helm to uninstall

• Determine helm release name from the resources found, e.g. based on labels heritage: Helm and release: <release-name>.
• Instead of removing individual resources, shell out to helm, executing helm uninstall <release-name>.

Some pros/cons:

• Pro: Applications installed by helm are probably best uninstalled by helm. There might be hooks, resources not annotated, etc. It would be consistent by design.
• Con: Major change in kube-janitor. Shell commands, helm binary requirement. Violates linux mantra 'do one thing well'.

Find helm release secret object and remove that as well

• Determine helm release name from the resources found, e.g. based on labels heritage: Helm and release: <release-name>.
• Helm release data would be in a secret having label name: <release-name> and a name like sh.helm.release.v1.<release-name>.v1.

Some pros/cons:

• Pro: Relatively easy to add to kube-janitor
• Con: Might leave resources that aren't annotated. This is already the case with current operation of kube-janitor but when using helm charts not authored yourself, you're dependent on the ability of the chart to annotate all resources. Note that this restriction doesn't seem to apply when using kube-janitor with a rules file. If I'm not mistaken that doesn't require having annotations on each resource.

Any implementation would obviously be 'opt in' and might require some additional config options or logic, e.g. an annotation specifying the label to extract the helm release from.

I'd like to hear your thoughts. Personally I think the 2nd approach would be a fit for kube-janitor while 1st approach has risk of embedding what would become a parallel completely new implementation.

Coming days (given time, 🎄after all) I'll try to run some tests to see how Helm copes with possible dangling resources, while release data has been removed.

[hjacobs]

My 2cts: I would like to avoid any Helm dependency if possible, but some additional logic for cleaning up Helm resources would be fine with me.

[runningman84]

What about cleaning custom resources used by Helm Operator? You would just have to delete some kubernetes object... helm itself would be called by the operator

[TBeijen]

As janitor iterates all api groups, cleaning those custom resources and then have Helm Operator do the rest, looks like a very good approach. :)

... now only if we used Helm Operator.

(Having explored the 'delete helm secret' approach a bit more: That will be tricky. Individual resources can expire at different times. You'd only want to delete the helm release secret if all associated resources are removed as well)

[sryabkov]

We could not use kube-janitor with helm 2 even though we were deploying everything to the same namespace. Having kube-janitor delete the namespace would leave helm releases still in the cluster. However, helm 3 changes this because helm releases now belong to a namespace instead of being global. Because of that, having kube-janitor support removal of helm releases is now less of an issue. By tweaking your Kubernetes namespace management, you can put kube-janitor annotation on the namespace, and that would delete the helm releases in that namespace when the namespace gets wiped out.

[hjacobs]

@sryabkov good to hear, I'm not a Helm user myself, but Helm 3 sounds like a step into the right direction :smile:

[bobo333]

The one caveat to the approach @sryabkov mentions, is non-namespaced resources in the helm release (ex: ClusterRoles) will not get deleted by deleting the namespace, not a huge problem but it's worth mentioning / being aware of.

[jspiro]

@sryabkov Have you tried this yet – does k-j delete the charts and completely wipe out the helm release? I.e. it should not show up in helm list after it clears the namespace.

If it doesn't work, what is missing from k-j?

[Stono]

The one caveat to the approach @sryabkov mentions, is non-namespaced resources in the helm release (ex: ClusterRoles) will not get deleted by deleting the namespace, not a huge problem but it's worth mentioning / being aware of.

This would be the biggest blocker for me, the purpose of janitor is to clean up your cluster, not leave things lying around :-)

I appreciate you don't want a helm dependency, but is used so extensively now - would it be such a bad thing to have it as an option?

Some use cases for us:

• We want applications which haven't been deployed in over 90 days to be auto purged from dev environments. The deployment could simply update the Janitor timestamp
• We have some ephemeral helm deployments used to debug clusters, which should be removed when people shut their session down but sometimes they're left around, purging them at midnight would be perfect

[sryabkov]

@jspiro We are using Helm 3, and Helm 3 releases are deployed to a particular namespace. Helm 3 stores all of the state/history data in Kubernetes secrets inside the namespace. If you delete the namespace, you delete all resources in the namespace, including all Helm 3 state data.

I am not sure what's going to happen if you have a cluster-wide resource in a chart and then you delete the namespace to which the chart was deployed (using Helm 3). My guess would be that the cluster-wide resource would still be around. We don't mix namespaced resources and cluster-wide resources in one chart.

[jspiro]

@sryabkov Awesome, that very encouraging. I'll be presenting a neat project in a CNCF webinar next month that partly relies upon helm/skaffold cleanup being complete. Stay tuned!