search

roboll / helmfile

Deploy Kubernetes Helm Charts
MIT License
4.05k stars 564 forks source link

[In the future] Helmfiles as a chart / Helmfile as a K8S CR #153

Open mumoshu opened 6 years ago

mumoshu commented 6 years ago

So that we can leverage everything helm provides to power helmfile :)

Points must be addressed before making it real:

faheem-nadeem commented 6 years ago

In favor for having a CRD. It could be something like a helm operator for a gitops style workflow, as in Flux from weaveworks. Which I find to be bit limiting at this moment.

mumoshu commented 5 years ago

I built a K8s operator for Helmfile CRD - https://github.com/mumoshu/helmfile-operator 😃Anyone interested in trying it out?

victornoel commented 5 years ago

I built a K8s operator for Helmfile CRD - https://github.com/mumoshu/helmfile-operator smileyAnyone interested in trying it out?

Nice, do you plan to make this a full-fledged component of the helmfile ecosystem and maintain it in the long term or is it just an experiment?

mumoshu commented 5 years ago

@victornoel Hey!

Honestly saying, I'm maintaining a bunch of projects including helmfile and helmfile-operator almost as an after-work hobby.

That's being said, I can't commit maintaining it in the long term until someone hires me for maintaining it, or perhaps make it a more collaborative project hosted in CFNF or anywhere else.

But I do have the "passion" to make it a fully-featured component that complements the whole helm eco-system.

I believe you'll suffer with (1)limited feedbacks(=harder to notice and debug deployment issues) and (2)the lack of a manual approval feature when helmfile is integrated with:

I've already written plans to address these in :

I tried hard to find solutions that works without putting too much engineering efforts into helmfile and helmfile-operator. But I have no way other than enhancing the helmfile-operator so far.

So I'd appreciate it if you could try it if you encountered the problem the operator is trying to solve, and give feedbacks and even better advocate for it, so that more people come and the project is likely to be maintained longer term.

The more a project attracts people, the more likely potential maintainers come :)

victornoel commented 5 years ago

@mumoshu sorry for taking so long to answer :)

After doing a bit of survey of existing solutions, I was wondering if it wouldn't be a better idea to integrate helmfile with the flux initiative. They just announced that they were going to join CNCF (or at least try) and detach themselves from weaveworks (in terms of governance): https://github.com/fluxcd/flux/wiki/MoveToFluxCD

mumoshu commented 5 years ago

@victornoel Thanks for the suggestion!

helmfile-operator is a tool comparable to helm-operator which is maintained as a part of flux. That is, I'm considering helmfile-operator as an effort to integrate helmfile into flux :)

Does it make sense to you?

Or are you suggesting to donate helmfile-operator into flux? If so, I'm fine to do so but not sure about the process and who actually want to maintain it if donated.

Any comment is welcomed.

victornoel commented 5 years ago

@mumoshu not donate no (well, except if you want to ^^), but yes, I was thinking about making something that would work well in the flux ecosystem. I'm not very knowledgeable about it, so I didn't realise that helmile-operator was already doing that :)

antaloala commented 3 years ago

@mumoshu @victornoel I just found this open issue. It was so nice for me to find it (I am some kind of new in helmfile -it is only some months I am playing with it, I am really impressed-) and I started to reason exactly as you did here above when trying to use helmfile in a GitOps scope. I think the new Flux v2 and the GitOps toolkit is really opening the door to move this helmfile-operator to Flux. (note old Flux v1 helm-operator was removed, replaced by a helm-controller that makes more sense); Flux v2 GitOps toolkit opens the door for new coming controllers to be added ... as the helmfile-controller ! It would be needed some integration towards the Flux v2/GitOps toolkit "source controller" (needing the HelmfileState CRD there?) to make it fully "usable" in Flux v2 What do you think?

mumoshu commented 3 years ago

@antaloala Hey! I've already read much about Flux v2 ecosystem and source-controller. I've also considered how I'd re-implement https://github.com/mumoshu/helmfile-operator in the Flux v2 ecosystem.

The question raised from my mind then was that- what's the concrete benefit of implementing it with source-controller?

As far as I can see, Flux v2 as a whole doesn't provide anything other than source-controller and the API to interact with Source from a custom controller perspective. For example, it would be great if they had support for nice Web UI like ArgoCD to interact/summarize Helmfile-related custom resources. But there aren't any.

Rewriting the helmfile-operator to use source does seem to allow me to reuse the some K8s controller and artifact fetching code from source-controller. But anything else?

I like Flux and Flux v2 and the new abstraction of Source/Artifact around source-controller. I'd really love to think about possibilities on how helmfile and flux v2 could enhance the whole u/x together. AFICS, there's no much benefit doing so, especially given we already have helmfile-operator.

WDYT?

victornoel commented 3 years ago

@mumoshu indeed, argocd seems clearly more full fledged than Fluxv2 in terms of available features. It seems one is really made with UIs in mind while the second is "just" description file-based gitops.

The only advantage of using FluxV2 over using nothing would be to get the community + all those various sources supported out of the box in a coherent ecosystem. I'm not sure it is worth the effort, as you say indeed. And we would get the same advantage from ArgoCD, both being under CNCF incubation btw.

Actually, the more I think about it, the best would even be that both ArgoCD and FluxV2 support helmfile (like they already support helm, kustomize, etc) and that you would have none or a minimum of work with maintaining it so that you can continue focusing on helmfile itself :)

Not sure what would be the best way towards this except maybe creating issues in those project to have them start thinking about it and realize the greatness of helmfile and the impossibility of not supporting it :D

mumoshu commented 3 years ago

@victornoel Hey! Thanks for sharing your ideas and insights.

FWIW, ArgoCD seems to provide us enough extension points to integrate Helmfile nicely.

A few months ago I've written some outdated recommendation on how you would integrate Helmfile with ArgoCD https://github.com/roboll/helmfile#argocd-integration. The gist of the recommendation was to run helmfile template on CI to generate manifests to git-commit, and let ArgoCD Application deplos the manifests in the generated git repo content.

What I've realized afterwards though was that more people use ConfigManagementPlugin.

With that, you point ArgoCD to the repo that contains no manifests, but helmfile.yaml and its siblings. You let ArgoCD run helmfile template on sync. It's like letting ArgoCD to run helmfile template | kubectl apply -f -.

It looks more like a native integration to me. What would be possible improvements over this ConfigManaegmentPlugin? 🤔

antaloala commented 3 years ago

@mumoshu, @victornoel I have not read enough about ArgoCD, so let me come back to the Flux v2 proposal. Yes, you are right, after some more reading on the GitOps toolkit provided by Flux v2 the integration with the source-controller is mainly about letting it to inspect the configured git repo and/or Helm repo to retrieve the required (from Helmfile-controller) artifacts. But I think it is not only that what we would get from a Flux v2 integrated Helmfile-controller, we would also have:

The main PROS is first bullet above; Helmfile is nowadays to be invoked periodically (a cron job running the "helmfile apply") and or coding/writing the instrumentation required to watch changes in a git repo (where the state files and the state value files are placed under git revision) and so trigger the "helmfile apply"; even when having a helmfile operator we still need "someone" to push into k8s API server the "helmfile" custom objects watched by the helmfile operator's custom controller. Without integrating that helmfile operator with a GitOps tool we cannot get that pull-based way of working.

But it is also true that once ArgoCD and Flux v2 provide support for Helm releases + allowing to connect different Helm releases across them (the "App of App" pattern in ArgoCD, in Flux v2/helm controller it is based on a HelmRelease object setting dependencies towards another HelmRelease object) it is not 100% clear to me if a GitOps integrated Helmfile operator is to be a long term solution or maybe some kind of gap filler as native helm-releases support in ArgoCD and Flux v2 evolve, getting more and more features.

What is your view on this? I must admit I do not see so clear now the long term value of having helmfile support in these CNCF GitOps projects ...

mumoshu commented 3 years ago

What is your view on this? I must admit I do not see so clear now the long term value of having helmfile support in these CNCF GitOps projects ...

@antaloala Hey! I believe having helmfile support would be great. I'm just saying that ArgoCD already supports it and it's great, and adding it to Flux v2 or enhancing helmfile-operator doesn't seem to work fully or worth the effort to me.

Flux v2/helm controller it is based on a HelmRelease object setting dependencies towards another HelmRelease object)

IIRC Flux v2's model doesn't allow cross-kind dependency. That is, a kustomize project can depend on another kustomize project and a helm release can depend on another helm release, but a kustomize project can not depend on a helm release. ArgoCD can handle this in another form, by using Sync Waves and Sync Phases.

Helmfile supports cross-kind dependency by needs.

But it is also true that once ArgoCD and Flux v2 provide support for Helm releases + allowing to connect different Helm releases across them (the "App of App" pattern in ArgoCD, in Flux v2/helm controller it is based on a HelmRelease object setting dependencies towards another HelmRelease object) it is not 100% clear to me if a GitOps integrated Helmfile operator is to be a long term solution or maybe some kind of gap filler as native helm-releases support in ArgoCD and Flux v2 evolve, getting more and more features.

As Flux v2 can't have cross-kind dependency, it's like comparing apple and orange. But anyway-

ArgoCD has out-of-box support for ConfigManagementPlugin, which allows you to integrate Helmfile into an arbitrary ArgoCD application. I think you can consider ArgoCD as it turns Application custom resources with config management plugins configured into something like a gitops operator for Helmfile.

ArgoCD Application + ConfigManagementPlugin looks great abstraction, allowing you to map an Application to a set of releases rendered by helmfile, or even map each Helmfile release to a dedicated ArgoCD Application.

This opens up possibilities for you to use your Helmfile configs for local development or standalone deployment, while using the same configs to power GitOps. This isn't possible with Helm or Kustomize alone.

You can even combine Helmfile-backed Application with Helm-backed and Kustomize-bakced Applications.

What else do we need beyond that? I'm curious 🤔

antaloala commented 3 years ago

@mumoshu Thanks for quick reply (as usual ;-) I was not really considering cases in which we need to cross-relate kustomize packaged (micro)services with Helm packaged ones in order to compose a cloud native application. Is that the normal case? I am focusing on scenarios in which a cloud native application is defined as a set of related helm releases (one per each instance of a required (micro)service), each required (micro)service providing its own Helm chart (plus some container images, of course). It is for this scenario where I think Hemfile, ArgoCD and Flux v2 overlap (all of them providing a fully declarative handling of helm releases).

I like (a lot) many of the additional feature you added to Helmfile + helm-x plugin (as the JSONpatches or StrategicMergePatches you can apply to a helm chart without having to fork it, the possibility to template the values.yaml to apply to each release, etc.) and so I thought on getting all of them in the two main CNCF GitOps projects: ArgoCD and Flux v2. But more I read on helm support (only helm) in ArgoCD and Flux v2, more I think they are aiming to cover Helmfile scope in a GitOps context ... but I could be totally wrong ...

mumoshu commented 3 years ago

Is that the normal case?

@antaloala Yeah it's a normal case for me. I usually use Helm charts for external projects whose official charts are distributed, and either a kustomize project or a local helm chart for my own thing. Helmfile manages all of them by just specifying chart value to either directory or a repo/chartname string, and writing needs to make a dependency graph.

I like (a lot) many of the additional feature you added to Helmfile + helm-x plugin (as the JSONpatches or StrategicMergePatches you can apply to a helm chart without having to fork it, the possibility to template the values.yaml to apply to each release, etc.) and so I thought on getting all of them in the two main CNCF GitOps projects: ArgoCD and Flux v2.

AFAIK, ArgoCD's ConfigManagementPlugin allows you to still leverage Helmfile features like JSONOPatches and StrageticMergePatches, because it basically works by injecting helmfile template command into ArgoCD's sync process. That's why I asked what else we might need.