search

seaweedfs / seaweedfs-csi-driver

SeaweedFS CSI Driver https://github.com/seaweedfs/seaweedfs
Apache License 2.0
220 stars 50 forks source link

Pod in Terminating State won't Unmount #118

Open washcycle opened 1 year ago

washcycle commented 1 year ago

Mounted storage can get into a state where the weed mount isn't unmounted when the pod is terminating.

I don't have a reproducibe example right now, but wanted to document the issue still.

Work around is to go into the k8s node and kill the pvc mount there.

root@aspen7:/# ps -aux | grep pvc-cebb2309-cdaa-4edd-a81d-bc75e6b5b15d
root     1334193  0.0  0.0 774944 54492 ?        Sl   May08   0:21 weed -logtostderr=true mount -dirAutoCreate=true -umask=000 -dir=/var/lib/k0s/kubelet/plugins/kubernetes.io/csi/seaweedfs-csi-driver/a28177b3447eebd81ee4b8be4305
7212220e657c2f1e8f7cbc696edbc3d8fb4f/globalmount -localSocket=/tmp/seaweedfs-mount-1226006495.sock -cacheDir=/var/cache/seaweedfs/pvc-cebb2309-cdaa-4edd-a81d-bc75e6b5b15d -collectionQuotaMB=25600 -concurrentWriters=15 -collectio
n=pvc-cebb2309-cdaa-4edd-a81d-bc75e6b5b15d -filer=seaweedfs-filer-client.seaweedfs:8888 -cacheCapacityMB=1024 -filer.path=/buckets/pvc-cebb2309-cdaa-4edd-a81d-bc75e6b5b15d

sudo kill 1334193

The pod will then terminate.

I wonder if there is something we can add to the csi driver that checks for terminating pods and will kill the seaweedfs mount, i'm not super good with the Go k8s stack, but perhaps inspiration from some other projects that have solved this would help.

chrislusf commented 1 year ago

Check whether NodeUnpublishVolume or NodeUnstageVolume is invoked in https://github.com/seaweedfs/seaweedfs-csi-driver/blob/master/pkg/driver/nodeserver.go

Maybe add some println.