Suggestion: Helmfile integration

[mumoshu]

Hi, thanks for sharing the great project!

Would you mind if I contribute a Helmfile integration as a deployment plugin to skaffold?

Helmfile is basically a declarative spec for deploying a set of helm charts from different chart repositories across namespace(s).

I believe it will nicely complements skaffold by providing additional benefits like

• Chart repositories management,
• Envvar interpolation
• Diffs

while allowing helmfile users to adopt skaffold for a higher-level of abstraction on top of helmfile. Note that, AFAIK, several company including Reddit adopted to it.

[dlorenc]

I like Helmfile, but with Helm 3 around the corner this one isn't going to be high up on our priority list.

[vietwow]

@dlorenc Dl Hi,

Why does Helm v3 relate to helmfile ?

VietNC

[balopat]

I'm going to close this for now due to lack of activity.

[iamsaso]

@mumoshu have you done any progress on this?

[shibumi]

Hi I would like to see this feature in skaffold, too.

[nkubala]

@shibumi if you had the ability to run an arbitrary pre-deploy script, would you be able to shell out to helmfile and do some necessary pre-processing, then let skaffold use helm to handle the rest?

realistically i'm not sure we'll be adding native helmfile support to skaffold any time soon, but we'll likely have pre/post- stage hooks added in the next 6 months.

[shibumi]

@nkubala I don't know how helmfile works in detail, but I don't think this will work. As far as I am aware helmfile does not generate a helm configuration. Helmfile directly calls helm as subprocess.

How difficult would be a PR for helmfile? I think I could try this, I contributed to other CNCF projects in the past.

[tejal29]

Thanks @shibumi, we are working on custom deploy script similar to Custom Build Script We are a bit hesitant of adding native helmfile support since we don't have a clear ownership model for community plugins.

However, #2277 could achieve the same. Would you be interested in writing a design proposal for that?

[tejal29]

closing this as #2277 could help address this.

[tonsV2]

I think it such a shame this was rejected. For a small app I often have a helm chart for the frontend, the backend and one for some kind of storage. With the use of helmfile I can keep track of which versions I have deployed to which environments.

I think the above setup is a fairly common and it would be great if it was supported by skaffold.

When I see comments as the one by @dlorenc I get the impression that the purpose of helmfile is somewhat misunderstood.

[shibumi]

@tonsV2 I think you misunderstood @dlorenc 's comment. @dlorenc said:

I like Helmfile, but with Helm 3 around the corner this one isn't going to be high up on our priority list.

He did not mean that Helmfile will be deprecated with Helm 3. He meant that the Helm 3 support for skaffold has higher priority than helmfile support for skaffold.

... at least that's how I read this.

[tonsV2]

@shibumi That's perfectly understandable and perhaps I'm lacking some insight but I don't see why this can't be developed in parallel.

My current issue is that I have values (username/password for the database) in my helmfile project which is shared by my database and my backend.

I love Skaffold but as soon as my projects grows beyond a single chart I can't use it without changing my code only for the sake of Skaffold and having uncommitted parts so it gets very messy and error prone.

[shibumi]

@shibumi That's perfectly understandable and perhaps I'm lacking some insight but I don't see why this can't be developed in parallel.

Uhm time? I guess they just have not enough man power to do it in parallel. Question is if they want to do it at all.. and the closed issue suggests otherwise. Would be interesting to know why.

[tonsV2]

@mumoshu wanted to contribute the plugin so I doubt that it's time

[nkubala]

we're trying to better understand Helm usage especially as it relates to skaffold, so I'm going to reopen this issue to continue the conversation.

in an ideal world we'd like to be able to extend our deployer interface to expose a spec which can be implemented by new (and potentially community-contributed) deployers. naturally I think if we're supporting Helm, then Helmfile could likely implement this spec.

@shibumi @tonsV2 or anyone else who wants to weigh in, a question - how important is Helm/Helmfile to the actual creation of resources in your cluster, as opposed to the templating of k8s manifests and packaging of charts? does Helm (or Helmfile) itself actually play an important role in the management of resources on the cluster?

our team is trying to understand separation of concerns between "rendering" (hydrating information into kubernetes resources) vs. "deploying" (actually sending this configuration to the k8s control plane to create resources in a cluster), especially as it relates to Helm.

[shibumi]

@nkubala Hi, thanks for reopening this issue.

We are deploying all of our resources via Helm/Helmfile. Our build chain looks as follows right now. We have dedicated project repositories. These repositories either contain one helm chart with a values file or they contain a helmfile for creating several helm charts at once. For this we have a self-build tekton pipeline that checks out each repository and triggers helm/helmfile directly.

There is no templating involved. We just call helmfile sync or helm upgrade ... directly.

[tonsV2]

Helmfile is to Kubernetes as docker-compose is to Docker. I use it to deploy a collection/stack of helm charts.

The credentials for those charts are interpolated from environment variables. I use direnv for the purpose.

So if I want to install the helm chart without helmfile I'll have to copy in the credentials thus have uncommitted, or ignored, files laying around in the project which leads to a mess and possible breaches.

I'm currently super busy so please excuse the limited amount of details in this post but I'm writing this so it's known that I would still appreciate Helmfile being supported by Skaffold.

[shibumi]

@tonsV2 thanks for your comment. I just want to add that the helmfile environments are the major advantage of helmfile over helm. With helm it is possible to have multiple value files, but managing these is very toilsome. With environments we are able to use gotemplating on different environments for different usecases (stage cluster, minikube, production cluster, different customers, etc). This was one of our main reasons why we chose helmfile for managing stacks of helm charts, because with helm charts we would start repeating ourselves in value files.

[tonsV2]

@shibumi Yes, with helmfile I can have different versions of my charts running in different environments which is a very fundamental and crucial use case.

The stack for my a small server of mine can be found here.

[shibumi]

@tonsV2 ah ok interesting. We go even one step further and we merge different environments in each other. This way we do not repeat the same values all the time that are being shared between environments.

[tonsV2]

@shibumi Do you have an example?

[shibumi]

@tonsV2 just add other value files to the list. Applied to your helmfile for example:

releases:
  - name: kubewatcher
    namespace: kubewatcher
    chart: tons/kubewatcher
    verify: true
    version: {{ .Values.kubewatcher.version }}
    installed: {{ .Values.kubewatcher.installed }}
    kubeContext: {{ .Values.kubeContext }}
    values:
      - values/base/kubewatcher-values.gotmpl
      - values/{{ .Environment.Name }}/kubewatcher-values.gotmpl

This will read two values files from the base environment and your current environment and merge them together.

[tonsV2]

@shibumi Thanks! Now I understand :-)

[tejal29]

5442 - related to

[obowersa]

Just to add a quick comment on the helmfile use cases. One of the reason's we're using it is to let us provide environment variables to helm as values, without needing to update the helm command for each environment within a pipeline, or generate a values file dynamically ( did consider this with jsonnet but was a little trickier to get adoption.)

Essentially the only differences between our environments ( both dynamic PR environments and 'static' ) environments are the environment variables set. Helmfile acts as a templating layer to overload the helm values.yaml with the appropriate environment variable values.

There's a bit more to our use case of it than that, but that's the main driver to our adoption of it. We also have some precedence ordering for base 'environment' + specific environment. For skaffold, I'd ideally be looking to support having a 'local/skaffold' env definition within helmfile.

Happy to go into more detail or provide some diagrams/use cases if it helps! At the moment we're still using skaffold with a legacy version of our helm setup before helmfile was introduced, but it's a bit of a maintence overhead. I've got a task in an upcomming sprint to look into how we can align them both a bit more.

---

Quick edit

We do use helmfile to then interact with the cluster as opposed to using helm directly, but that's more of an indirect thing.

[fzyzcjy]

Hi, is there any updates?

[nkubala]

@fzyzcjy and others, update from the team: we're actively working through a redesign of the way we handle "rendering" of kubernetes resources vs. creation of these resources on a cluster, detailed in this design proposal. these changes should make it more clear how Helmfile should interact with Skaffold's resource model, so once we ship those changes we can explore concretely how to integrate with the Helmfile toolchain.

[addshore]

I ended up reading a bunch of issues on this repo relating to helm and helmfile usage of skaffold last night and ended up getting to a point that worked quite well for my setup.

I wrote this up here with example and talking through the various gotchas that got me and also my reading of https://github.com/GoogleContainerTools/skaffold/issues/2277 https://github.com/GoogleContainerTools/skaffold/issues/3497 https://github.com/GoogleContainerTools/skaffold/issues/2631 which other folks might encounter.

The TLDR here is that I used a hook

    hooks:
      before:
        - host:
            command: [ "./helmfile-values", "-e", "local", "-r", "ui", "-n" ]

That does something like this:

helmfile --environment $TARGET_ENVIRONMENT --selector name=$TARGET_RELEASE write-values --skip-deps --output-file-template $OUTPUT_YAML
yq eval 'del(.image)' --no-colors $OUTPUT_YAML > $OUTPUT_YAML.altered
mv $OUTPUT_YAML.altered $OUTPUT_YAML

And then use the resulting temporary file in skaffold

        valuesFiles:
          - ".tmp.values.ui.yaml"
          - NeverPull.yaml

Hopefully this helps someone else also going down this path.

[tejal29]

Thanks you documenting your usecase and approach. I am going to close this as won't fix. Please use hooks as suggested by @mumoshu

[addshore]

Thanks you documenting your usecase and approach. I am going to close this as won't fix. Please use hooks as suggested by @mumoshu

cough by @addshore ;)