delete deployment should clean all data & external resources -avoids leaks

poblin-orange commented 3 years ago

Expected behavior

As a k3s-boshrelease operator

In order to avoid leaking Iaas/3rd party resources (such as disk, load balancers, ...)
I need all resources to be deleted before a k3S cluster gets deleted

As a k3s-boshrelease operator

In order to avoid leaking confidential data stored in k8s resources (such as K8S secrets) when a k3s cluster is deleted
I need all confidential data to be deleted before a k3S cluster gets deleted

Observed behavior

Upon k3S deployment deletion,

[ ] the K8S resources are not deleted
[ ] the k3s agent and server files are not explicitly deleted, including mounted K8S secrets

Alternative solutions

[ ] Use bosh deployment lifecycle hook: pre-stop with BOSH_DEPLOYMENT_NEXT_STATE=delete

If BOSH_DEPLOYMENT_NEXT_STATE is set to delete then one can safely conclude that consequently both instance and its VM will also be deleted. Similarly, when BOSH_INSTANCE_NEXT_STATE is set to delete then the corresponding VM also will be deleted after the stop process. The pre-stop script also uses an exit code to indicate its success (exit code 0) or failure (any other exit code).

[ ] delete CR resources

[ ] trigger k8s cluster lifecycle hooks to ask for clean up ?

[ ] Study industry K8S cluster deletion behaviors

[x] GKE

[x] https://cloud.google.com/kubernetes-engine/docs/how-to/deleting-a-cluster

When you delete a cluster, the following resources are deleted:

The control plane resources
All of the node instances in the cluster
Any Pods that are running on those instances
Any firewalls and routes created by GKE at the time of cluster creation
Data stored in host hostPath and emptyDir volumes

The following resources are not deleted:

External load balancers created by the cluster
Internal load balancers created by the cluster
Persistent disk volumes

[x] GKE workarounds: Use Iaas API to delete volumes annotated with cluster id https://stackoverflow.com/questions/59655519/deleting-all-the-associate-persistent-disks-when-deleting-cluster

Use GCP API to delete volumes annotated with cluster id

> Cloud SDK can be used to identify the disks if the proper filter and format are parse > i.e. > To list all the disks being used by a GKE (you can change the filter at your convenience) > ``` > gcloud compute disks list --format="table(name,users)" --filter="name~^gke-" > ``` > To list only disks used as PVC > ``` > gcloud compute disks list --format="table(name,users)" --filter="name~^gke-.*-pvc-.*" > ``` > This last command will list detached PVC disks > ``` > gcloud compute disks list --format="table(name,users)" --filter="name~^gke-.*-pvc-.* AND -users:*" > ``` > To ensure a detached disk is not in use by a cluster, here's a kubectl command to list a cluster's PVs and their GCE PD: > ``` > kubectl get pv -o custom-columns=K8sPV:.metadata.name,GCEDisk:spec.gcePersistentDisk.pdName > ``` >

[ ] explicitly delete all K8S resources
- [x] ~~Using kubectl delete nodes on each node~~: merely deregister workers from the master according to stackoverflow question
- [ ] sequentially delete each namespace
  - [ ] This may hang/fail during finalizer errors, see https://www.openshift.com/blog/the-hidden-dangers-of-terminating-namespaces. Might be wise to fail the bosh pre-stop hook and leave it up to an operator to diagnose and fix.
- [ ] sequentially delete cluster-wide resources. The list could to be filtered from output of $ kubectl api-resources --namespaced=false:

$ kubectl api-resources --namespaced=false

``` $ kubectl api-resources --namespaced=false NAME SHORTNAMES APIGROUP NAMESPACED KIND componentstatuses cs false ComponentStatus namespaces ns false Namespace nodes no false Node persistentvolumes pv false PersistentVolume mutatingwebhookconfigurations admissionregistration.k8s.io false MutatingWebhookConfiguration validatingwebhookconfigurations admissionregistration.k8s.io false ValidatingWebhookConfiguration customresourcedefinitions crd,crds apiextensions.k8s.io false CustomResourceDefinition apiservices apiregistration.k8s.io false APIService tokenreviews authentication.k8s.io false TokenReview selfsubjectaccessreviews authorization.k8s.io false SelfSubjectAccessReview selfsubjectrulesreviews authorization.k8s.io false SelfSubjectRulesReview subjectaccessreviews authorization.k8s.io false SubjectAccessReview certificatesigningrequests csr certificates.k8s.io false CertificateSigningRequest clusterissuers certmanager.k8s.io false ClusterIssuer nodes metrics.k8s.io false NodeMetrics ingressclasses networking.k8s.io false IngressClass runtimeclasses node.k8s.io false RuntimeClass podsecuritypolicies psp policy false PodSecurityPolicy clusterrolebindings rbac.authorization.k8s.io false ClusterRoleBinding clusterroles rbac.authorization.k8s.io false ClusterRole priorityclasses pc scheduling.k8s.io false PriorityClass csidrivers storage.k8s.io false CSIDriver csinodes storage.k8s.io false CSINode storageclasses sc storage.k8s.io false StorageClass volumeattachments storage.k8s.io false VolumeAttachment ```

[ ] clean up k3S resources
- [ ] Using k3S uninstall scripts on agents and servers
  - [ ] Possibly inline uninstall script commands see https://github.com/k3s-io/k3s/blob/cb73461a5bd69cc91f3dc5fec3ffbbd8e970132a/install.sh#L640-L690

gberche-orange commented 3 years ago

Expected behavior

As a k3s-boshrelease operator
In order to avoid leaking Iaas/3rd party resources (such as disk, load balancers, ...)
I need all resources to be deleted before a k3S cluster gets deleted

Observed behavior

Upon k3S deployment deletion, the K8S resources are not deleted

Alternative solutions

[ ] Use bosh deployment lifecycle hook: pre-stop with BOSH_DEPLOYMENT_NEXT_STATE=delete

If BOSH_DEPLOYMENT_NEXT_STATE is set to delete then one can safely conclude that consequently both instance and its VM will also be deleted. Similarly, when BOSH_INSTANCE_NEXT_STATE is set to delete then the corresponding VM also will be deleted after the stop process.
[ ] trigger k8s cluster lifecycle hooks to ask for clean up
[ ] explicitly delete all resources

poblin-orange commented 3 years ago

Interesting to see how an integrated k8s runtime like gke handles these leaks access. However, on a generic k8s runtime like k3s cant exactly know which type of external resource could leak (with out of tree Cloud Provider, container based storage like longhorn, custom ingress / loadbalancer / externalDNS).

Probably a security mechanism could be set (preventing deletion if a list of configurable kinds is still present, blocking deletion).

With this, the cleanup of these objects should be let to upper level provisionning layer. see:

orange-cloudfoundry / k3s-wrapper-boshrelease