Releases for hybrid installations

[yulianedyalkova]

We already have installations like wallaby that are hybrid. Is there anything that needs to be done in a different way than the single-provider ones? Any technical limitations?

[AverageMarcus]

app-admission-operator has been updated to handle cross-provider Releases in https://github.com/giantswarm/app-admission-controller/pull/547 (forgot to link to this issue)

I still need to work on converting the Releases repo to make use of Flux as a GitRepo rather than deploying as an app-collection. Need to perform some testing (likely on glippy) and figure out where would make most sense to implement it.

[AverageMarcus]

Reminder on how to go about migrating resources from Apps to GitOps without removing the resources:

1. Apply app-operator.giantswarm.io/paused: "true" as an annotation on the App CR
2. Apply chart-operator.giantswarm.io/paused: "true" as an annotation on the Chart CR
3. Apply the GitRepo and Kustomization resources to the cluster that will take over managing the resources
4. Delete the App and Chart CRs for the releases.

[Gacko]

Reminder on how to go about migrating resources from Apps to GitOps without removing the resources:

1. Apply app-operator.giantswarm.io/paused: "true" as an annotation on the App CR
2. Apply chart-operator.giantswarm.io/paused: "true" as an annotation on the Chart CR
3. Apply the GitRepo and Kustomization resources to the cluster that will take over managing the resources
4. Delete the App and Chart CRs for the releases.

As also noted in the Slack thread, this will leave the Helm release resources on the cluster. Rather use the helm.sh/resource-policy: keep annotation. https://helm.sh/docs/howto/charts_tips_and_tricks/#tell-helm-not-to-uninstall-a-resource

[AverageMarcus]

So, what's needed for supporting hybrid installations is:

• We need to migrate to using GitOps and Flux for deploying the releases rather than deploying them as App CRs (partially covered by https://github.com/giantswarm/roadmap/issues/3663)
• To achieve the above, we must first remove our use of the Kustomization plugin (see https://github.com/giantswarm/releases/pull/1458)
• We need to pause the App platform from reconciling the release Apps anymore (see above two comments)
• We should add a new extra directory to management-cluster-bases that contains the GitRepo and Kustomization needed for each provider's releases (e.g. ./extras/releases/capv/)
• We should then update the *-management-clusters Kustomization.yaml for each CAPI MC to include a new resource entry for each provider that MC will support creating WCs for that points to the new bases directories as mentioned above.
• We would need to update mc-bootstrap (I think) to ensure these releases are references via the Kustomization.yaml rather than as an app.
• We need to remove the releases app from the app catalogs, remove the CircleCI job that pushes to the catalog and then finally manually clean up the Release CRs to remove the extra labels and annotations leftover from when it was managed by Helm.

---

For reference, this is the GitRepo and Kustomizations I used when testing this approach on glippy. These are the kind of thing that should be included in the new management-cluster-bases directories.

apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: capi-releases
  namespace: flux-giantswarm
spec:
  interval: 30s
  ref:
    branch: master
  url: https://github.com/giantswarm/releases

---

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: vsphere-releases
  namespace: flux-giantswarm
spec:
  force: false
  interval: 5m
  path: ./vsphere
  prune: false
  sourceRef:
    kind: GitRepository
    name: capi-releases
    namespace: flux-giantswarm
  retryInterval: 1m
  timeout: 3m

---

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: azure-releases
  namespace: flux-giantswarm
spec:
  force: false
  interval: 5m
  path: ./azure
  prune: false
  sourceRef:
    kind: GitRepository
    name: capi-releases
    namespace: flux-giantswarm
  retryInterval: 1m
  timeout: 3m