Allow manifest deployer annotations to change behavior of managing the resources lifecycle

onedr0p commented 7 months ago

Is your feature request related to a problem? Please describe.

Currently the manifest deployer will manage the lifecycle of user based manifests added to the /var/lib/k0s/manifests dir:

By default, k0s reads all manifests under /var/lib/k0s/manifests and ensures that their state matches the cluster state. Moreover, on removal of a manifest file, k0s will automatically prune all of it associated resources.

The use of Manifest Deployer is quite similar to the use the kubectl apply command. The main difference between the two is that Manifest Deployer constantly monitors the directory for changes, and thus you do not need to manually apply changes that are made to the manifest files.

In my usecase I would like to use the manifest deployer to deploy a kube-vip daemonset and rbac and then have Flux "take over" the manifests deployed there however this is currently not possible.

Describe the solution you would like

It would be cool if there was a annotation we could set on each manifests that would change this behavior. Something like this:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: kube-vip
  namespace: kube-system
  annotations:
    # If set, will not prune if files are removed from disk:
    deployer.k0sproject.io/prune: Disabled
    # If set, will ignore after applied once:
    deployer.k0sproject.io/once: Enabled

Describe alternatives you've considered

There doesn't seem to be any alternative since k0s wants to assume complete ownership of the lifecycle of user added manifests to this directory.

Additional context

Having this ability would be great for GitOps based clusters, since we can check in manifests to Git after they are applied by k0s and manage the lifecycle outside of editing files on the nodes disk.

jnummelin commented 7 months ago

In your use case, why can't you manage them from GitOps for the first time? In other words, why the first apply is via manifest deployer?

onedr0p commented 7 months ago

This is because I am deploying kube-vip as the "external" Kube API LB which is really a core component to the cluster. In my use-case it's nice to have this come up with the cluster as opposed to after with Flux so I can use the kube-vip addr right away in my kubeconfig before Flux comes online.

onedr0p commented 7 months ago

It would be great if the helm extension also had this functionality. I use renovate bot to keep the k0sctl config updated and from what I read here it looks like if any value changes in the charts it will override what Flux or Argo has deployed when I go to use k0sctl apply again.

...
        extensions:
          helm:
            repositories:
              - name: cilium
                url: https://helm.cilium.io
            charts:
              - name: cilium
                chartname: cilium/cilium
                # renovate: datasource=github-releases depName=cilium/cilium
                version: 1.15.0-rc.1
                namespace: kube-system
                values: |2
                  autoDirectNodeRoutes: true
...

Basically I am hoping for a way to deploy infra workloads (cilium/kube-vip) using k0sctl and/or the manifests deployer and NOT have it be maintained by k0s after they are installed, it would be nice to only have k0s install them once and not deleted if removed or changed in the k0sctl config file or k0s directories. This would have a great user experience for people using GitOps who want to deploy infra workloads with k0s native methods and then have Flux or Argo take them over.

Side note: I am able to do this with k3s by annotating the k3s HelmChart CR for Cilium with a annotation and then deleting it so Flux can take it over. It would be great if k0s had something like this too.

twz123 commented 7 months ago

Basically I am hoping for a way to deploy infra workloads (cilium/kube-vip) using k0sctl and/or the manifests deployer and NOT have it be maintained by k0s after they are installed, it would be nice to only have k0s install them once and not deleted if removed or changed in the k0sctl config file or k0s directories.

This is a very valid and typical use case when managing clusters, e.g. via GitOps. Do the bare minimum and go from there with the tools of your choice. The Manifest Deployer and to some extent the Helm extension are probably not the right tools for this. They're essentially themselves ultra-minimalist, bare-bones Flux/Argo CD variants. While we could probably find a solution for this, via annotations or something, I'd be a bit reluctant to add it to the current codebase. This is mainly for the reasons given in https://github.com/k0sproject/k0s/pull/3854#issuecomment-1876989215.

For the use case you have in mind, k0sctl might be a better option. It could get a feature to apply arbitrary additional manifests during k0sctl apply, as a one-shot operation. That'd be much simpler™.

Side note: I am able to do this with k3s by annotating the k3s HelmChart CR for Cilium with a annotation and then deleting it so Flux can take it over. It would be great if k0s had something like this too.

I could imagine that k0s could allow to bypass Helm release uninstallations on custom resource deletions if the helm.k0sproject.io/uninstall-helm-release finalizer is not present on the resource.

k0sproject / k0s