search

vanvalenlab / kiosk-console

DeepCell Kiosk Distribution for Kubernetes on GKE and AWS
https://deepcell-kiosk.readthedocs.io
Other
35 stars 6 forks source link

Update the make helmfile/destroy/all command for helm3 #389

Closed willgraf closed 3 years ago

willgraf commented 4 years ago

helm3 updates the helm delete command to require the namespace of the helm deployment, which breaks our current implementation. Fortunately, helmfile destroy will delete all installed helm charts regardless of namespace, and is a much simpler command than piping each name/namespace to helm destroy.

This issue was not discovered during the helm 3 migration as the make command only deleted the deepcell namespace, the other pods were destroyed during cluster destruction and did not have any PVCs that got stranded.

willgraf commented 4 years ago

This PR is blocked by a prometheus-operator==8.12.3 issue when deleting CRDs installed by prometheus operator. This should be resolved by #317.

willgraf commented 4 years ago

The issue is NOT resolved. Despite prometheus-operator being upgraded, helmfile destroy does NOT successfully delete it:

Error: uninstallation completed with 1 error(s): unable to build kubernetes objects for delete: [unable to recognize "": no matches for kind "Alertmanager" in version "monitoring.coreos.com/v1", unable to recognize no matches for kind "PrometheusRule" in version "monitoring.coreos.com/v1", unable to recognize "": no matches for kind "Prometheus" in version "monitoring.coreos.com/v1", unable to recognize "": no matches for kind "PrometheusRule" in version "monitoring.coreos.com/v1", unable to recognize "": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1"]

I believe a manual workaround involves turning off cleanupCustomResource and manually deleting the CRDs.

There is more work required in figuring out how to reliably delete the helm charts during cluster tear-down.

willgraf commented 3 years ago

This issue is being worked on in another PR #406