search

cloudfoundry / cf-for-k8s

The open source deployment manifest for Cloud Foundry on Kubernetes
Apache License 2.0
300 stars 115 forks source link

cf-for-k8s fails to delete on k8s 1.18 (gcp and eks) #573

Open paulcwarren opened 3 years ago

paulcwarren commented 3 years ago

Describe the bug

On two different k8s 1.18 distros (gcp and eks) we have seen a failure to delete cf-for-k8s because the database PV fails to delete.

To Reproduce*

Steps to reproduce the behavior:

  1. Deploy a k8s 1.18 cluster on gcp/eks
  2. Deploy cf-for-k8s
  3. kapp delete -a cf
  4. Observe that the deletion fails because the postgres db pv cannot be unbound

Expected behavior

The kapp deletion is successful

cf-for-k8s SHA

Latest develop

Cluster information

Observed so far on 1.18 GCP & EKS

cf-gitbot commented 3 years ago

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/175746818

The labels on this github issue will be updated when the story is started.

jamespollard8 commented 3 years ago

As you can see in https://github.com/cloudfoundry/cf-for-k8s/commit/72988bc38bfe5993ad6e31b0938259b5384ca9ca, we found a relatively simple workaround. We just need to force delete the pod (after first deleting the statefulset backing it) to get kapp delete working even on newer versions of Kubernetes.

kubectl delete statefulset cf-db-postgresql -n cf-db
kubectl delete pod cf-db-postgresql-0 -n cf-db --force --grace-period 0

We'll do some more investigation to figure out a path forward towards a proper solution to this problem but for now, we're at least unblocked and now running the latest K8s versions for our EKS and GKE tests.