helm + krane

[kfox1111]

¹ _{Helm is already well established in the community and has a wealth of prebuilt packages. It would be great if krane could be used with helm so that you use helm to install your software but use krane to determine the success of the rollout.}

[KnVerey]

Something that we've often considered but not yet implemented is a krane watch command that would take a deploy directory and monitor all the corresponding live resources. The use cases we had in mind were resuming a watch after a deploy times out, and debugging the performance of krane itself. However, I'm thinking that such a command could also serve your use case. Do you agree?

[kfox1111]

That might work, but still needs a way to have krane know what to watch. I can think of a couple of ways to maybe handle that. Maybe have krane look at all objects in k8s with a particular label? That might work with all k8s deployment methods (kustomize, ksonnet, etc). Alternately, maybe the 'helm template' command could be used to generate the deploy directory. That does run a slight risk of the helm template command using flags not matching what was deployed.

[aladmit]

I already use krane with helm for like a half of year. My way is use helm charts for everything, render it by helm template and past it to krane through pipe. So I have all power of helm charts and krane knows what to deploy and watch, because it gets a regular yaml to input. It works perfect even in cases where helm chart uses helm dependencies.

I have only one problem with this way of using krane. It doesn't work at all when you want to deploy a huge community helm chart like Prometheus Operator which uses a several namespaces and krane fails with error, cause its force to use only 1 namespaces at a time. In this case I just deploy chart by pure helm.

[kfox1111]

I'm interested in the helm lifecycle management though, which helm template does not work with.

IE, helm ls, helm upgrade, helm delete, etc.

krane does not work with all of helm, just statically prerendered helm templates.

What helm kind of lacks, is 'helm watch until fail/succeed' and more imperative flows. It has some rudimentary support but the last few times I've tried it, it has not been very reliable.

krane looks like it could very nicely bridge the gap between helm's lifecycle management with ci/cd runners.

[KnVerey]

That might work, but still needs a way to have krane know what to watch. I can think of a couple of ways to maybe handle that. the 'helm template' command could be used to generate the deploy directory.

This is the MVP for a krane watch IMO: it would have similar arguments to krane deploy (but with namespace being optional if a selector is given, so that resources from global deploys could also be watched).

Maybe have krane look at all objects in k8s with a particular label?

Yeah, though it should keep the mental model from our other commands. i.e. assume we want to target either a namespace, a selector-based partition of a namespace, or a selector-based partition of the global namespace (never multiple namespaces). We could use the last-applied annotation to figure out which resources krane manages.

This option would be simpler to use and more flexible, but would be hairier to implement and would run more slowly, especially in large namespaces. That's because it would require crawling all the resource kinds available and fetching all the instances in scope to find out if we created them or not. We'd also have to decide what to do with the case where the user running the watch doesn't have RBAC for some kinds (do we assume there aren't any instances / they aren't ours, or throw an error like the deploy would?). I'd be open to having this as an option, but the directory-based mode should come first.

Are you interested in contributing by any chance? 😄

[kfox1111]

Good point about knowing what to watch by having to crawl everything. I hadn't considered that.

Could still work but yeah, would be slower/harder on the apiserver. Especially when CRD's are also becoming more widely used. You'd have to crawl through all of those too...

So then, maybe more engine specific things may be better...

I'm not sure how reliable it is, but in helm 3, it looks like the entire uploaded state for the release, including all prerendered objects, are stored now:

kubectl get secret sh.helm.release.v1.gitlab-runner-operator.v1 -n gitlab-runner-operator -o go-template='{{.data.release | base64decode | base64decode}}' | zcat | jq -r .manifest
<snip> yaml multipart doc with all Kinds rendered.

what if we had a 'krane helm-release which acted similarly to what @aladmit suggested. But instead of manually calling helm template and dumping it to the directory, the command would into a temporary directory. And then it would short circuit the actually uploading of the objects and then continues on watching just the defined objects?

Alternately, it could maybe be done as two commands piped together.... helm-release-fetch | krane watch where the first part does the kubectl get secret / extract .manifest from above to stdout, and krane watch expects in a set of yaml with all things to watch without ever uploading?

[KnVerey]

Alternately, it could maybe be done as two commands piped together.... helm-release-fetch | krane watch

This makes sense to me. helm-release-fetch wouldn't be written on this repo though. Although we're very eager to be composable with other Kubernetes tools, we're not really interested in maintaining wrappers for specific ones ourselves.

[dudicoco]

Hi,

Any updates on this? Currently i'm using https://github.com/werf/kubedog to watch my helm deployments, would be happy to try out krane for that purpose as well.