Best way to customized the rke2 charts and avoid having default manifest written

[zifeo]

Is your feature request related to a problem? Please describe.

A HelmChart can only have a single HelmChartConfig. Because of this, the following situation is hard to manage well:

• deploy via IaC a custom version of the helm chart that overrides the manifests deployed on launch
• let the cluster owner update the config later (which clashes with IaC)

Describe the solution you'd like

Multiple HelmChartConfig (not this repo) or (better) a way to repackage the HelmChart before they get written to the folder.

Describe alternatives you've considered

A manual script after the manifest are deployed in the folder, however this leaves a space where the old ones are alive and introduce a point of failure post launch. Restarting the server will also reset them.

Additional context

[brandond]

I'm confused by what you're asking for. Are you trying to use external tooling to manage the chart deployments? Or are you just asking for a way to feed even more values content into the charts?

Can you explain what exactly you're doing that makes the current combination of HelmChart + HelmChartConfig not usable? Why do you need a 3rd/4th/Nth values source?

[zifeo]

@brandond Rather the latter.

To give a concrete example: the clusters we manage are flavored = they only come with cilium as CNI and without kube-proxy, so we need to make an initial configuration and this responsibility is given to the platform team. However each cluster may need ad-hoc configurations like enabling Hubble on that chart and this is left to the team owning the cluster. This means that we need something like HelmChart + HelmChartConfig (base conf that is enforced) + HelmChartConfig (ad-hoc conf based on that cluster specific need) with IaC management.

Currently, we are patching the HelmChart on the start but this is causing some inconvenient race conditions between the manifest. Does that give a clearer picture?

[brandond]

Could you perhaps work with your platform team to allow the teams that own the cluster some mechanism to inject additional YAML ValuesContent into the HelmChartConfig that the platform team is deploying?

HelmChartConfigs were intended to be the way for administrators to influence the configuration of packaged charts; if you have multiple teams managing that I'm not sure the correct answer is to push that problem upstream and allow an arbitrary number of HelmChartConfig resources for a given chart, or to allow injection of content into the packaged HelmChart itself.

[zifeo]

There is not always a direct link (open source) between the platform team and the teams owning the cluster. The "interface" between them is basically a VM image which does not facilitate collaboration. Could the RKE2 package be repackaged with modified manifests or that would require to rebuild everything?

[brandond]

The chart manifests are in the rke2-runtime image. If you wanted to modify the packaged HelmCharts you'd have to build your own modified version of that image, and point at it via the runtime-image config flag.

There is a unique runtime image for each rke2 release where the tag corresponds to the RKE2 version; v1.30.1+rke2r1 for example uses docker.io/rancher/rke2-runtime:v1.30.1-rke2r1.

[brandond]

Normally an ask of this size would go through our product management team for evaluation and prioritization. The current mechanism of using matching names for the HelmChart and HelmChartConfig resources doesn't lend itself well to allowing for arbitrary additional values sources.

Maybe we could add something like support for an ValuesSecret field on the HelmChart/HelmChartConfig spec, but it would still be on your platform team to set that field in the HelmChartConfig that they are deploying in order to enable that if we were to add it.

[zifeo]

Thanks @brandond! Let me first try with runtime-image: docker.io/custom-rancher/rke2-runtime. For the HelmChartConfig, the perfect solutions would be to have multiple ones with a fixed order? https://github.com/k3s-io/helm-controller/issues/180

[brandond]

You need to point it at a tag (not the implicit :latest), and you'll need to rebuild the image on top of the new tag and update the value whenever you upgrade RKE2, or you'll continue to use binaries and charts from the old version of RKE2.

I commented on that issue as to why I don't like the requested approach. I don't think any approach that allows for an arbitrary number of additional resources to read values from is a good idea. The furthest I think we'd want to go is a ValuesSecret field that allows specifying the name of a Secret in the same namespace as the HelmChart/HelmChart resources, as a source for additional values content. This would align well with other fields (DockerRegistrySecret/AuthSecret) that read user-provided content from optional user-provided Secrets.

[brandond]

If you have cycles for it, we'd be glad to review a PR to helm-controller to add that behavior.