search

kubernetes-retired / external-storage

[EOL] External storage plugins, provisioners, and helper libraries
Apache License 2.0
2.69k stars 1.6k forks source link

[efs-provisioner] Fails to delete "saved" volume #1314

Closed Nuru closed 4 years ago

Nuru commented 4 years ago

Using efs-provisioner v2.4.0 on Kubernetes 1.15.10, I noticed it fails to delete the storage for a PersistentVolume if the PersistentVolume was previously "saved" after being deleted externally.

Using efs StorageClass with

reclaimPolicy: Delete
volumeBindingMode: Immediate

When a PersistentVolumeClaim (pvc) is created using the efs storage class, efs-provisioner creates a PersistentVolume (pv) and stores the data on the EFS file system in a directory named /efs-provisioner/{pvc-name}-{pv-name}. When the PersistentVolumeClaim is deleted and the reclaimPolicy is "Delete", efs-provisioner deletes its PersistentVolume and removes the /efs-provisioner/{pvc-name}-{pv-name} directory and all the data in it from the EFS file system.

If the PersistentVolume Kubernetes resource is deleted, for example via kubectl delete, efs-provisioner will attempt to "save" it and the logs will show something like:

Trying to save persistentvolume "pvc-bbe5e4fa-d9e3-4a43-aed7-d9f3a4f31cfe"
persistentvolume "pvc-bbe5e4fa-d9e3-4a43-aed7-d9f3a4f31cfe" saved

The bug is that after saving a PersistentVolume this way, if you then immediately (and maybe later) delete the PersistentVolumeClaim, efs-provisioner deletes the PersistentVolume Kubernetes resource, but it does not remove the data from EFS. It is abandoned there and difficult to notice or clean up using standard Kubernetes tools. Since EFS storage is relatively expensive, this can be a significant problem.

fejta-bot commented 4 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

Nuru commented 4 years ago

/remove-lifecycle stale

Can someone please look into this?

nikhita commented 4 years ago

Thanks for reporting the issue!

This repo is no longer being maintained and we are in the process of archiving this repo. Please see https://github.com/kubernetes/org/issues/1563 for more details.

If your issue relates to nfs provisioners, please create a new issue in https://github.com/kubernetes-sigs/nfs-ganesha-server-and-external-provisioner or https://github.com/kubernetes-sigs/nfs-subdir-external-provisioner.

Going to close this issue in order to archive this repo. Apologies for the churn and thanks for your patience! :pray: