Volume mount inside Pod stuck when no more disk space

[spastorclovr]

• To reproduce:

  • create a pod with a 10Gi seaweedfs-class volume claim attached and mount it under /data
  • within the pod , create a file using dd above 10Gi

• What is happening

  • dd fails with an error related to disk space
  • df shows correctly the space being 100% used
  • running : ls /data in the pod will hang indefinitively

• Additional info

  • Nothing special in logs from seaweed-csi pods
  • Trying to delete the pod will hang as well with error killing pod: [failed to "KillContainer" for "my-frontend" with KillContainerError: "rpc error: code = DeadlineExceeded desc = context deadline exceeded", failed to "KillPodSandbox" for "bb5cd03d-52c3-4fa5- 9b20-750be4b7bd92" with KillPodSandboxError: "rpc error: code = DeadlineExceeded desc = context deadline exceeded"
  • A force delete would work. From a new pod, the listing of /data will still hang
  • From the filer webui, the file will appear to have a size of 0

Weed version:

version 30GB 3.61 8ae00e47a linux amd64

Images used:

image: registry.k8s.io/sig-storage/csi-provisioner:v3.5.0 image: registry.k8s.io/sig-storage/csi-resizer:v1.8.0 image: registry.k8s.io/sig-storage/csi-attacher:v4.3.0 image: registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.8.0 image: registry.k8s.io/sig-storage/livenessprobe:v2.10.0 image: chrislusf/seaweedfs-csi-driver:latest

[chrislusf]

What is the expected behavior?

[spastorclovr]

Well ... the expected behaviour would be to still be able to list a mounted volume without being stuck (at least to be able to cleanup space for instance); like this is a case for any local mount being full.

[spastorclovr]

Update: it seems to happen when there is only one file occupying all the available space. When creating multiple small (10MB) files, all is well : nothing got stuck.