Spike: OCIOps

[manno]

Just like GitRepo for GitOps, we want a OCI/Helm resource in Fleet to deploy Helm charts in a more direct way. Ideally this leverages OCI to the fullest.

Research

• how do other projects handle OCI, e.g. is there a combined manifest for helm chart + image, or are this always two OCI repositories?
• The OCIOps resource would contain configuration and targeting/customization rules
• this spike does not need to deal with OCI artefact mirroring, assume the OCI registry is reachable from upstream and downstream
• should work with common registries like harbor and public docker, ghcr,
• polling and version scanning don't need to be in the first iteration, reacting on a change to the OCIRepo resource is enough

[Danil-Grigorev]

This feature will be of a great value for the native integration with Cluster API. Cluster API currently leverages project https://github.com/kubernetes-sigs/cluster-api-addon-provider-helm/ which provides resource like HelmChartProxy to deploy helm charts into CAPI clusters using generated kubeconfig for the Cluster. This resource serves as a template for helm chart installation in the cluster, and allows to manage full lifecycle of the chart.

General use-case for them, is to allow installing arbitrary workloads with a per-cluster modifications to the payload from a Helm source, where the state of the Helm chart is stored in the API server. This can be used in CNI and CCM installations for the cluster, as shown by the Calico example.

Overall requirements are described in the proposal: https://github.com/kubernetes-sigs/cluster-api/blob/main/docs/proposals/20220712-cluster-api-addon-orchestration.md, but the proposal does not cover GitOps story, allowing a room for improvement in this area as per alternatives section.

Upon initial investigation, a rough equivalent for the HelmChartProxy is the Bundle resource in Fleet, but the spec.helm section is not processed in a non GitOps based environment.

[Danil-Grigorev]

A rough equivalent for the calico HelmChartProxy resource is a Bundle resource, which has CAPI Cluster resource templating provided under ClusterValues automatically.

apiVersion: fleet.cattle.io/v1alpha1
kind: Bundle
metadata:
  name: calico-cni
spec:
  defaultNamespace: default
  helm:
    releaseName: calico
    repo: https://docs.tigera.io/calico/charts
    chart: tigera-operator
    values:
      installation: |-
        cni:
          type: Calico
          ipam:
            type: HostLocal
        calicoNetwork:
          bgp: Disabled
          mtu: 1350
          ipPools: ${- range $cidr := .ClusterValues.Cluster.spec.clusterNetwork.pods.cidrBlocks }
          - cidr: "${ $cidr }"
            encapsulation: None
            natOutgoing: Enabled
            nodeSelector: all()${- end}
  targets:
    clusterName: capi-quickstart

which produced correct BundleDeployment and is easy to grasp as it is 1:1 replica to the original, but is not functional due to limitation, requiring usage of fleet as a CLI to perform resource unpacking.

spec:
  deploymentID: s-ca3d163bab055381827226140568f3bef7eaac187cebd76878e0b63e9e442:fb6456765448626792c6beefc2ed116bf84cbb5e695072f715bf2a79b6f4695b
  options:
    defaultNamespace: default
    helm:
      chart: tigera-operator
      releaseName: calico
      repo: https://docs.tigera.io/calico/charts
      values:
        installation: |-
          cni:
            type: Calico
            ipam:
              type: HostLocal
          calicoNetwork:
            bgp: Disabled
            mtu: 1350
            ipPools:
              cidr: "10.1.0.0/16"
              encapsulation: None
              natOutgoing: Enabled
              nodeSelector: all()