Open onedr0p opened 7 months ago
In your use case, why can't you manage them from GitOps for the first time? In other words, why the first apply is via manifest deployer?
This is because I am deploying kube-vip as the "external" Kube API LB which is really a core component to the cluster. In my use-case it's nice to have this come up with the cluster as opposed to after with Flux so I can use the kube-vip addr right away in my kubeconfig before Flux comes online.
It would be great if the helm extension also had this functionality. I use renovate bot to keep the k0sctl config updated and from what I read here it looks like if any value changes in the charts it will override what Flux or Argo has deployed when I go to use k0sctl apply
again.
...
extensions:
helm:
repositories:
- name: cilium
url: https://helm.cilium.io
charts:
- name: cilium
chartname: cilium/cilium
# renovate: datasource=github-releases depName=cilium/cilium
version: 1.15.0-rc.1
namespace: kube-system
values: |2
autoDirectNodeRoutes: true
...
Basically I am hoping for a way to deploy infra workloads (cilium/kube-vip) using k0sctl and/or the manifests deployer and NOT have it be maintained by k0s after they are installed, it would be nice to only have k0s install them once and not deleted if removed or changed in the k0sctl config file or k0s directories. This would have a great user experience for people using GitOps who want to deploy infra workloads with k0s native methods and then have Flux or Argo take them over.
Side note: I am able to do this with k3s by annotating the k3s HelmChart
CR for Cilium with a annotation and then deleting it so Flux can take it over. It would be great if k0s had something like this too.
Basically I am hoping for a way to deploy infra workloads (cilium/kube-vip) using k0sctl and/or the manifests deployer and NOT have it be maintained by k0s after they are installed, it would be nice to only have k0s install them once and not deleted if removed or changed in the k0sctl config file or k0s directories.
This is a very valid and typical use case when managing clusters, e.g. via GitOps. Do the bare minimum and go from there with the tools of your choice. The Manifest Deployer and to some extent the Helm extension are probably not the right tools for this. They're essentially themselves ultra-minimalist, bare-bones Flux/Argo CD variants. While we could probably find a solution for this, via annotations or something, I'd be a bit reluctant to add it to the current codebase. This is mainly for the reasons given in https://github.com/k0sproject/k0s/pull/3854#issuecomment-1876989215.
For the use case you have in mind, k0sctl might be a better option. It could get a feature to apply arbitrary additional manifests during k0sctl apply
, as a one-shot operation. That'd be much simpler™.
Side note: I am able to do this with k3s by annotating the k3s HelmChart CR for Cilium with a annotation and then deleting it so Flux can take it over. It would be great if k0s had something like this too.
I could imagine that k0s could allow to bypass Helm release uninstallations on custom resource deletions if the helm.k0sproject.io/uninstall-helm-release
finalizer is not present on the resource.
Is your feature request related to a problem? Please describe.
Currently the manifest deployer will manage the lifecycle of user based manifests added to the
/var/lib/k0s/manifests
dir:In my usecase I would like to use the manifest deployer to deploy a kube-vip daemonset and rbac and then have Flux "take over" the manifests deployed there however this is currently not possible.
Describe the solution you would like
It would be cool if there was a annotation we could set on each manifests that would change this behavior. Something like this:
Describe alternatives you've considered
There doesn't seem to be any alternative since k0s wants to assume complete ownership of the lifecycle of user added manifests to this directory.
Additional context
Having this ability would be great for GitOps based clusters, since we can check in manifests to Git after they are applied by k0s and manage the lifecycle outside of editing files on the nodes disk.