State Mgmt Templates MVP

Goals

Ensure that clusters created by the cluster mgmt component of hmc can also be easily be operated as actual k8s clusters (as just a k8s cluster itself does not provide much value for the business and also needs additional helmcharts/artifacts installed) with bringing a simple State Management solution which does not need any additional dependencies installed.
While there are many state mgmt solutions out there that allow extremely complex configurations and systems we believe that for standard services like ingress-nginx a simpler solution based on the existing templates can be built. This allows platform teams to simply create and manage many clusters without the need to learn a complex state management solution like Flux or Argo.

Major deliverables

Implement a state mgmt solution which allows platform admins to also define helmcharts in a Template which then automatically get installed in clusters after they have been provisioned.

Who it benefits

Customer Business: Faster time to market for clusters, less overhead of the platform team to manage 10 or 1000 clusters
Platform: Central place to define what should be installed across multiple clusters without the need to duplicate the same configuration for every cluster
Mirantis: Provide customers with a State Mgmt solution which is multi cluster capable if the customer does not have this capability yet, therefore providing a better service and experience

Acceptance criteria

Existing Templates can not only define which CAPI objects should be installed, but also possibility to define helmcharts that should be installed in the cluster after the cluster has been provisioned
Initially we just need ingress-nginx as examples for installing services through a template, more services will be added later.
Templates have possibilities to overwrite values which then get injected into the helm charts installed in the clusters (as it is the case for cluster mgmt)
leverage flux HelmRelease CAPI capability to install the helm charts defined in the template into the clusters
Platform Engineers don't see any indications that we are using flux HelmRelease under the hood, it is abstracted for them (as we are abstracting CAPI for the cluster mgmt). People creating templates do need to learn how to use Flux HelmRelease
Updating a helmchart in a template will automatically roll out the changes to the clusters created from that template based on the Upgrade/Rollout System
State Management solution is end to end tested via an Github CI solution

Assumptions

While there is a solution by the CAPI community currenty being built called CAAPH, we are using Flux HelmRelease already and therefore are using the Flux HelmRelease CAPI capability to reduce complexity for ourselves.
Today the templates leverage the k0s helm controller to install infrastructure specific helm charts, we agreed we leave these helm charts inside of k0s and only add additional helmcharts via HelmRelease
We are using only inline Values of HelmRelease and no Values references

Telemetry & Success Criteria

Out of scope

Using Flux GitOps approach, we just use the Flux HelmRelease capability
- Looking into full Flux GitOps will happen later, as the current capabilities does not scale well if a customer has 20+ templates
Deploy a single HelmChart across multiple Templates, we just need HelmCharts deployed through an existing Template

Related issues:

Mirantis / hmc