Provide support for CRD upgrades natively in zarf

mjnagel commented 3 months ago

Is your feature request related to a problem? Please describe.

Today zarf's posture for CRD upgrades is identical to helm's - https://docs.zarf.dev/ref/deploy/#handling-customresourcedefinitions-crds. While this does make sense given zarf is using helm under the hood it means that the end user is left to figure out how to upgrade CRDs on their own. Typically this means either the chart needs to handle upgrades (I've seen this done with k8s jobs that apply the CRD manifests) or it has to be handled via the zarf package. Handling via the zarf package is certainly possible, but not ideal (I think it would have to be done with files + actions, since even manifests get wrapped in a helm chart and would rely on helm's behavior).

Describe the solution you'd like

I would like zarf to provide config options for CRDs per chart. By default this would align with the upstream helm default behavior (install only, don't touch upgrades), but could be configured based on package needs. This could mean something like a new field in the helm chart component spec to specify whether to do CRD upgrades or not:

components:
  - name: my-chart
    charts:
      - name: my-chart
        url: https://mychart.com
        version: 2.0.0
        namespace: myns
        crdUpgrade: true

When this field is set Zarf would handle CRD upgrades itself (this would have to be done via k8s api/sdk directly rather than helm given the helm posture on CRDs).

Describe alternatives you've considered

Continue down the current path, shifting the CRD upgrade responsibility to the end user to handle either via the helm chart or some zarf package changes. Perhaps to ease the burden on end users and example package could be provided to show how CRD upgrades would be expected to take place.

Additional context

Another project in a similar space of "managing helm installs" is Flux - they have build a similar spec around their HelmRelease which allows for configuring CRD upgrades. I'm not sure if their project/code would be a useful place to look for inspiration on how to approach this problem differently.

I do think CRD upgrades are likely to be a common problem/need in the future - many upstream charts leave this to the end user as a manual upgrade (ex: kube-prom-stack). It would be great if zarf could provide something to lessen this burden.

phillebaba commented 3 months ago

I think we should just follow a similar method to what Flux does. I was partially involved in the discussions at the time, and it became clear that managing CRDs was required. Implementing this should be possible for Zarf as we already have the chart so getting the CRDs from the crd directory in the chart should be simple. We should also keep to Flux choice to not delete CRDs and only create or update them.

mjnagel commented 3 months ago

@phillebaba yeah I think that would be a great path to go down. One thing I was always curious on was the decision to only "manage" CRDs if they are in the crd directory. In my experience with a number of the opensource charts we use for uds-core there's definitely a handful of charts that use templates for CRDs instead. I wonder if it would possible/reasonable to use the kind only (and not consider directory) to determine what is a CRD we want to manage?

phillebaba commented 1 month ago

Here is a user example that does not involve the Helm crd directory.

kind: ZarfPackageConfig
metadata:
  name: knative
  description: Deploys Knative using Manifest files
  version: 0.0.1
  authors: Justin Bailey
  documentation: https://knative.dev/docs/install/yaml-install/serving/install-serving-with-yaml/
  source: https://github.com/knative/serving

components:
  - name: knative-serving
    required: true
    description: "Knative Serving"
    manifests:
      - name: knative-crds
        files:
          - https://github.com/knative/serving/releases/download/knative-v1.14.1/serving-crds.yaml
      - name: knative-serving
        namespace: knative-serving
        kustomizations:
          - manifests/
    images:
      - gcr.io/knative-releases/knative.dev/serving/cmd/activator:v1.14.1
      - gcr.io/knative-releases/knative.dev/serving/cmd/autoscaler:v1.14.1
      - gcr.io/knative-releases/knative.dev/serving/cmd/controller:v1.14.1
      - gcr.io/knative-releases/knative.dev/serving/cmd/queue:v1.14.1
      - gcr.io/knative-releases/knative.dev/serving/cmd/webhook:v1.14.1
      - gcr.io/knative-releases/knative.dev/net-kourier/cmd/kourier:v1.14.0
      - docker.io/envoyproxy/envoy:v1.26-latest

zarf-dev / zarf