[FEAT] - Cluster off-boarding automation

IvoGoman commented 5 months ago

Priority

(Low) Something is a little off

Description

In order to ensure no objects are left behind there should be a garbage collection that removes all Greenhouse managed resources from the managed cluster. Under normal circumstances the Greenhouse Controller will ensure that their managed CRs are removed during the Lifecycle.

There should be a safe-guarding measure in place to avoid accidental deletion of a cluster.

Cluster AdmissionWebhook denies direct deletion but labels the Cluster to indicate the deletion intend
Once a cluster is marked as deleted the reconciliation of all resources should stop. Except for the automatic renewal of the KubeConfig, as to ensure that the cluster can still be accessed if the deletion label is removed again.
Status of the Cluster show indicate that it is not-ready and marked for deletion. To make this visible in UI/CLI
After 48h Greenhouse controller will delete the cluster. Deletion will only complete once all resources deployed by Greenhouse are removed (Propagated Resources, Team RBAC CRs, Installed Plugins) The deletion of these is triggered by the Controller.

Acceptance criteria:

[ ] Cluster cannot be deleted right away but follows above
[ ] Cluster has no more Greenhouse resources after being deleted from Greenhouse
[ ] e2e test for deleting a Cluster and ensuring propagated resources, RBAC and installed plugins are removed. #292
[ ] document this behaviour

Reference Issues

No response

IvoGoman commented 3 months ago

This should provide a mechanism for Greenhouse controllers to add their CRDs to this GC mechanism/ prevent the Cluster from being deleted before the resources were deleted.

auhlig commented 1 month ago

Under normal circumstances the Greenhouse Controller will ensure that their managed CRDs are removed during the Lifecycle.

The other mechanisms sound great, but CRD removal is risky. Deleting the CR definition triggers garbage collection of all its instances. A plugin integrates natively with the customer cluster and users might use CRDs outside of Greenhouse. IMHO Greenhouse should follow the Helm logic and not delete CRDs. WDYT @IvoGoman?

IvoGoman commented 1 month ago

@auhlig small but important typo. It should of course be CR instead of CRD.

cloudoperators / greenhouse