search

uselagoon / remote-controller

A group of controllers for handling Lagoon builds and tasks in Kubernetes or Openshift
5 stars 1 forks source link

AKS: during decomissioning of lagoon enviornment: delete all PVCs before namespaces #50

Closed Schnitzel closed 3 years ago

Schnitzel commented 3 years ago

Azure AKS has a strange but, when you delete a namespace it tries to delete the PVCs in it, but unfortunately a secret that is used to provision the PVCs is deleted before the PVCs itself and so the provisioner of the PVC cannot remove it (slowclap).

Therefore we need to: If we are on AKS and we're decommissioning a Lagoon Environment: Remove all Deployments/Statefulset/Replicasets/Deamonsets (which removes all Pods), then remove the PVCs, wait for all PVCs to be removed, then remove the PVC

Schnitzel commented 3 years ago

More background on the issue: https://winterdom.com/2018/07/26/kubernetes-azureFile-dynamic-volumes-deleting

Schnitzel commented 3 years ago

Internal Ticket reference: https://control.vshn.net/tickets/AMZE-3023

Schnitzel commented 3 years ago

@shreddedbacon According to VSHN this still happens, I did some tests on test9 and when deleting a namespace that has a PVC for the nginx deployment this was in the logs:

2021-07-20T18:33:59.074Z    INFO    handlers.LagoonTasks.Deletion   Received remove task for project drupal-example-test9, branch main - drupal-example-test9-main
2021-07-20T18:33:59.187Z    INFO    handlers.LagoonTasks.Deletion   Deleted deployment nginx in  drupal-example-test9-main for project drupal-example-test9, branch main
2021-07-20T18:33:59.200Z    INFO    handlers.LagoonTasks.Deletion   Deleted deployment cli in  drupal-example-test9-main for project drupal-example-test9, branch main
2021-07-20T18:33:59.540Z    INFO    handlers.LagoonTasks.Deletion   Deleted namespace drupal-example-test9-main for project drupal-example-test9, branch main

there is no deletion of the PVC happening, I think this is probably because we list PVs instead of PVCs: https://github.com/amazeeio/lagoon-kbd/blob/55d0a6667787ca83e964db381e8292145bac6923/handlers/delete_handler.go#L153 ?