Add `ensure` Option to `spec.uninstall` for Ensuring Proper Cleanup of HelmChartProxy Resources

[kahirokunn]

User Story

As an operator, I would like an option to ensure the uninstallation of HelmChartProxy resources. This is to properly clean up external resources during cluster deletion and prevent unnecessary resource remnants or unexpected behavior.

Detailed Description

I propose adding a new option, ensure, to the spec.uninstall section of HelmChartProxy to control its uninstallation behavior. By enabling this option, specific Helm charts can be guaranteed to uninstall properly during cluster deletion. The default value of ensure should be false, allowing users to explicitly opt into this behavior.

An example configuration:

apiVersion: addons.cluster.x-k8s.io/v1alpha1
kind: HelmChartProxy
metadata:
  name: karpenter
spec:
  clusterSelector:
    matchLabels:
      karpenterChart: enabled
  repoURL: oci://public.ecr.aws/karpenter
  chartName: karpenter
  options:
    waitForJobs: true
    wait: true
    timeout: 5m
    install:
      createNamespace: true
    uninstall:
      ensure: true  # New option (default: false)
  valuesTemplate: |
    controller:
      replicaCount: 2

Challenges

Currently, the following issues can arise during cluster deletion, making proper cleanup difficult:

1. Cluster Resource Deletion Before HelmChartProxy Uninstallation
   If the Cluster resource is deleted before HelmChartProxy has been fully uninstalled, external resources created by the Helm chart may remain orphaned, leading to inconsistencies and potential security risks.

2. Concurrency Between HelmChartProxy Uninstallation and Cluster Deletion
   When the Cluster deletion process starts while the HelmChartProxy is still uninstalling, the Cluster deletion may execute too quickly, preventing the HelmChartProxy from completing its uninstallation process. This can result in incomplete cleanup of resources.

To address these challenges, I suggest leveraging the Cluster API [Runtime Hooks for Add-on Management](https://github.com/kubernetes-sigs/cluster-api/blob/main/docs/proposals/20220414-runtime-hooks.md). Specifically, the BeforeClusterDelete hook can be used to:

• Ensure that HelmChartProxy resources are fully uninstalled before proceeding with Cluster deletion.
• Introduce synchronization points to prevent race conditions between HelmChartProxy uninstallation and Cluster resource deletion.

Other Information

• Using Runtime Hooks can integrate the add-on uninstallation process with the cluster lifecycle, ensuring a smoother deletion process.
• For Helm charts that create external resources (e.g., Karpenter), this feature guarantees resource consistency and proper cleanup.
• Addressing these challenges reduces the risk of resource remnants, inconsistencies, and security issues during cluster lifecycle management.

/kind feature

[Jont828]

/triage-accepted

Will take a look on Thurs.