search

datashim-io / datashim

A kubernetes based framework for hassle free handling of datasets
http://datashim-io.github.io/datashim
Apache License 2.0
479 stars 68 forks source link

Uninstall (kubectl delete -f) stuck after "dlf-mutating-webhook-cfg" deleted #252

Open juan-sg opened 1 year ago

juan-sg commented 1 year ago

Summary:kubectl delete command gets stuck after mutatingwebhookconfiguration.admissionregistration.k8s.io "dlf-mutating-webhook-cfg" deleted (which seems to be the last declaration in the YAML).

Series of events: This was working as expected (although documentation can be a bit better given that the s3 bucket[s] do[es]n't work OOTB) until I tried to switch from a stringData secret to just data. This resulted in the PVC getting stuck on 'terminating' status, so I decided to delete the PV and PVC. PVC did not get deleted, so I issued kubectl patch pvc example-dataset -p '{"metadata":{"finalizers":null}}' and the PVC was removed but creating the dataset deployment again would not create the PVC this time. I decided to uninstall the whole thing and reinstall, but the kubectl delete -f command got stuck after mutatingwebhookconfiguration.admissionregistration.k8s.io "dlf-mutating-webhook-cfg" deleted.

The entire stdout is found below.

kubectl delete -f https://raw.githubusercontent.com/datashim-io/datashim/master/release-tools/manifests/dlf.yaml
namespace "dlf" deleted
serviceaccount "csi-attacher-nfs" deleted
serviceaccount "csi-nodeplugin" deleted
serviceaccount "csi-s3" deleted
serviceaccount "csi-attacher" deleted
serviceaccount "csi-provisioner" deleted
serviceaccount "dataset-operator" deleted
secret "webhook-server-tls" deleted
storageclass.storage.k8s.io "csi-s3" deleted
customresourcedefinition.apiextensions.k8s.io "datasetsinternal.com.ie.ibm.hpsys" deleted
customresourcedefinition.apiextensions.k8s.io "datasets.com.ie.ibm.hpsys" deleted
clusterrole.rbac.authorization.k8s.io "external-attacher-runner-nfs" deleted
clusterrole.rbac.authorization.k8s.io "csi-nodeplugin" deleted
clusterrole.rbac.authorization.k8s.io "csi-s3" deleted
clusterrole.rbac.authorization.k8s.io "external-attacher-runner" deleted
clusterrole.rbac.authorization.k8s.io "external-provisioner-runner" deleted
clusterrole.rbac.authorization.k8s.io "dataset-operator" deleted
clusterrolebinding.rbac.authorization.k8s.io "csi-attacher-role-nfs" deleted
clusterrolebinding.rbac.authorization.k8s.io "csi-nodeplugin" deleted
clusterrolebinding.rbac.authorization.k8s.io "csi-s3" deleted
clusterrolebinding.rbac.authorization.k8s.io "csi-attacher-role" deleted
clusterrolebinding.rbac.authorization.k8s.io "csi-provisioner-role" deleted
clusterrolebinding.rbac.authorization.k8s.io "dataset-operator" deleted
role.rbac.authorization.k8s.io "external-attacher-cfg" deleted
role.rbac.authorization.k8s.io "external-provisioner-cfg" deleted
rolebinding.rbac.authorization.k8s.io "csi-attacher-role-cfg" deleted
rolebinding.rbac.authorization.k8s.io "csi-provisioner-role-cfg" deleted
service "csi-attacher-nfsplugin" deleted
service "csi-attacher-s3" deleted
service "csi-provisioner-s3" deleted
service "webhook-server" deleted
daemonset.apps "csi-nodeplugin-nfsplugin" deleted
daemonset.apps "csi-s3" deleted
deployment.apps "dataset-operator" deleted
statefulset.apps "csi-attacher-nfsplugin" deleted
statefulset.apps "csi-attacher-s3" deleted
statefulset.apps "csi-provisioner-s3" deleted
csidriver.storage.k8s.io "ch.ctrox.csi.s3-driver" deleted
mutatingwebhookconfiguration.admissionregistration.k8s.io "dlf-mutating-webhook-cfg" deleted

Please let me know if there are any additional details needed to troubleshoot this. Thanks!

srikumar003 commented 1 year ago

@juan-sg Apologies for late reply. kubectl delete getting stuck after the last line means that there is a Datashim managed object still around in the cluster.

What could've happened is that the PVC got deleted but, for some reason, the associated dataset and datasetinternal objects were not. The following commands will confirm this: $ kubectl get datasets --all-namespaces $ kubectl get datasetsinternal --all-namespaces

In case, there are Datasets/DatasetInternal objects still around, you can patch them the same way you did for the PVC, and they would go away (along with the namespace and the rest of the install)

Also, noted the suggestion to improve documentation. if you could list here what you expected and what was missing, we'll update accordingly.