search

ceph / ceph-csi

CSI driver for Ceph
Apache License 2.0
1.28k stars 545 forks source link

snapshot space is not reclaimed by ceph after snapshot delete #4855

Closed Borosan closed 1 month ago

Borosan commented 1 month ago

reclaim space on ceph after deleting unwanted snapshots

deleted snapshots uses space in ceph cluster!

Environment details

Steps to reproduce

Steps to reproduce the behavior:

  1. setup a k8s cluster 1.27
  2. install an external ceph cluster
  3. install kubernetes external snapshotter
  4. instal ceph-csi on k8s
  5. check for pvc creation and snap shot using ceph-csi example.

Actual results

everything works fine but ceph cluster doesn't free up space after snapshot removing. describe:

  1. create a pvc
  2. create a pod
  3. take snapshot1
  4. add data to pod
  5. take snapshot2
  6. add data to pod
  7. take snapshot 3
  8. check for ceph pool used space
  9. restore to snapshot1
  10. delete snapshot 2,3
  11. check for ceph pool used space. ---> still in used after even 24 hours

Expected behavior

ceph pool used space decrease

Logs

Nothing special, everything works fine on kubernetes side and ceph side

k8s (ceph-csi): I0916 11:24:08.342006 1 controller.go:1337] provision "ceph-csi-rbd/rbd-pvc" class "csi-rbd-sc": started I0916 11:24:08.343467 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"ceph-csi-rbd", Name:"rbd-pvc", UID:"2b5b07e0-e529-4b0d-826b-b84834b67b83", APIVersion:"v1", ResourceVersion:"36135484", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "ceph-csi-rbd/rbd-pvc" I0916 11:24:08.498841 1 controller.go:1442] provision "ceph-csi-rbd/rbd-pvc" class "csi-rbd-sc": volume "pvc-2b5b07e0-e529-4b0d-826b-b84834b67b83" provisioned I0916 11:24:08.498868 1 controller.go:1455] provision "ceph-csi-rbd/rbd-pvc" class "csi-rbd-sc": succeeded I0916 11:24:08.507966 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"ceph-csi-rbd", Name:"rbd-pvc", UID:"2b5b07e0-e529-4b0d-826b-b84834b67b83", APIVersion:"v1", ResourceVersion:"36135484", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-2b5b07e0-e529-4b0d-826b-b84834b67b83 I0916 11:34:40.902443 1 controller.go:1471] delete "pvc-2b5b07e0-e529-4b0d-826b-b84834b67b83": started E0916 11:34:41.020746 1 controller.go:1481] delete "pvc-2b5b07e0-e529-4b0d-826b-b84834b67b83": volume deletion failed: rpc error: code = Internal desc = rbd csi-vol-27e28c84-ac90-435d-bb83-a590ca751212 is still being used W0916 11:34:41.020785 1 controller.go:989] Retrying syncing volume "pvc-2b5b07e0-e529-4b0d-826b-b84834b67b83", failure 0 E0916 11:34:41.020822 1 controller.go:1007] error syncing volume "pvc-2b5b07e0-e529-4b0d-826b-b84834b67b83": rpc error: code = Internal desc = rbd csi-vol-27e28c84-ac90-435d-bb83-a590ca751212 is still being used I0916 11:34:41.020839 1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolume", Namespace:"", Name:"pvc-2b5b07e0-e529-4b0d-826b-b84834b67b83", UID:"bdf48e95-7aa6-4e79-b2f4-808f960c59dc", APIVersion:"v1", ResourceVersion:"36138490", FieldPath:""}): type: 'Warning' reason: 'VolumeFailedDelete' rpc error: code = Internal desc = rbd csi-vol-27e28c84-ac90-435d-bb83-a590ca751212 is still being used I0916 11:34:41.521419 1 controller.go:1471] delete "pvc-2b5b07e0-e529-4b0d-826b-b84834b67b83": started I0916 11:34:41.661569 1 controller.go:1486] delete "pvc-2b5b07e0-e529-4b0d-826b-b84834b67b83": volume deleted I0916 11:34:41.678707 1 controller.go:1531] delete "pvc-2b5b07e0-e529-4b0d-826b-b84834b67b83": persistentvolume deleted I0916 11:34:41.678740 1 controller.go:1536] delete "pvc-2b5b07e0-e529-4b0d-826b-b84834b67b83": succeeded

when we notice at ceph . it create a a chain of volumes (link each child to its parent) but when snap shot is deleted . some volumes (snaps) are deleted but no difference is seen in pool used space . it seem ceph some how keeps all data.

Madhu-1 commented 1 month ago

@Borosan rbd snapshots/clone doesn't occupy extra storage and deleting snapshots also doesn't free up the space (unless they are flattened), This is expected behavior. please clarify why you think the space consumption should decrease. Do you see space consumption increase when you take the snapshot?

Borosan commented 1 month ago

@Madhu-1 thank you for response. we did the test again : 1- create a pvc 2- create a pod 3- generate some data (5G) 4-take snap-1 5-generate some data (5+5) on pod 6-take snap-2 7- generate data on po (5+5+5) 8-take snap-3 then we create a restore pvc from snap-2 and create a restore pod next we deleted snapshot from kuber and also first pvc . the images (lets say vol and snp) removed from ceph . the restored pod had data from snap-2 (5+5) but there was 0 used spaced on ceph . we flattened the vol on ceph and it shows real used data which was 10G(5+5) seems ok! but the question is there are 1000 of pods and several pvc,pv , snapshots . we might take several snap shots on each namespace (customer) and move forward or reverse . what is the automatic mechanism to handle the volumes which needs to be flattened ? should we track pv,pvc , snap and related volumes on ceph for each namespace (customer)? notice that flattened feature is already enabled on our pool. is there something like a cron job ? or we should create something? or enable it?

Rakshith-R commented 1 month ago

CephCSI automatically takes care of flattening when certain depth limit (parent-child-child ) or horizontal limit(snaps/pvc pvc clone on same parent pvc) is reached. Users don't need worry about internal details.

Rakshith-R commented 1 month ago

Please use the discussion section for further clarifications. Thanks