rbd: add volume locks for reclaimspace operations

[iPraveenParihar]

Describe what this PR does

This commit adds VolumeLocks on reclaimspace operations to prevent multiple process executing rbd sparsify/fstrim on same volume.

Related issues

Closes: #4637

Checklist:

• [x] Commit Message Formatting: Commit titles and messages follow guidelines in the developer guide.
• [x] Reviewed the developer guide on Submitting a Pull Request
• [x] Pending release notes updated with breaking and/or notable changes for the next major release.
• [x] Documentation has been updated, if necessary.
• [x] Unit tests have been added, if necessary.
• [x] Integration tests have been added, if necessary.

---

Show available bot commands

These commands are normally not required, but in case of issues, leave any of the following bot commands in an otherwise empty comment in this PR: * `/retest ci/centos/`: retest the `` after unrelated failure (please report the failure too!)

[iPraveenParihar]

/test ci/centos/mini-e2e/k8s-1.28

[iPraveenParihar]

Created a PVC of 1Gi and wrote dummy data of 500M and deleted it.

root@rbd-pod-test:/# dd if=/dev/zero of=/var/lib/www/html/dummy-file bs=1M count=500
500+0 records in
500+0 records out
524288000 bytes (524 MB, 500 MiB) copied, 1.6972 s, 309 MB/s
root@rbd-pod-test:/# sync
root@rbd-pod-test:/# rm /var/lib/www/html/dummy-file
root@rbd-pod-test:/# ls -l /var/lib/www/html/
total 16
drwx------ 2 root root 16384 May 29 07:23 lost+found

before ReclaimSpace operation -

[root@c1 /]# rbd du -p replicapool
warning: fast-diff map is not enabled for csi-vol-58a49f13-ff2e-4415-b1a8-21ec59d95017. operation may be slow.
NAME                                          PROVISIONED  USED
csi-vol-58a49f13-ff2e-4415-b1a8-21ec59d95017        1 GiB  528 MiB

created ReclaimSpaceJob for the PVC

pm@dhcp53-176:~$ cat <<EOF | k create -f -
> apiVersion: csiaddons.openshift.io/v1alpha1
kind: ReclaimSpaceJob
metadata:
  name: sample-1
spec:
  target:
    persistentVolumeClaim: rbd-pvc-test
  backOffLimit: 10
  retryDeadlineSeconds: 900
  timeout: 600
> EOF
reclaimspacejob.csiaddons.openshift.io/sample-1 created

/NodeReclaimSpace - fstrim operation succeed

I0529 07:29:28.134875 2566810 utils.go:198] ID: 23 GRPC call: /reclaimspace.ReclaimSpaceNode/NodeReclaimSpace
I0529 07:29:28.138386 2566810 utils.go:199] ID: 23 GRPC request: {"secrets":"***stripped***","staging_target_path":"/var/lib/kubelet/plugins/kubernetes.io/csi/rook-ceph.rbd.csi.ceph.com/cff6345ad1aff4cb25770de65b8012a934530bfd9d29bc77c557332df5607126/globalmount","volume_capability":{"AccessType":{"Mount":{}},"access_mode":{"mode":7}},"volume_id":"0001-0009-rook-ceph-0000000000000008-58a49f13-ff2e-4415-b1a8-21ec59d95017"}
I0529 07:29:28.439310 2566810 cephcmds.go:105] ID: 23 command succeeded: fstrim [/var/lib/kubelet/plugins/kubernetes.io/csi/rook-ceph.rbd.csi.ceph.com/cff6345ad1aff4cb25770de65b8012a934530bfd9d29bc77c557332df5607126/globalmount/0001-0009-rook-ceph-0000000000000008-58a49f13-ff2e-4415-b1a8-21ec59d95017]

[root@c1 /]# rbd du -p replicapool
warning: fast-diff map is not enabled for csi-vol-58a49f13-ff2e-4415-b1a8-21ec59d95017. operation may be slow.
NAME                                          PROVISIONED  USED
csi-vol-58a49f13-ff2e-4415-b1a8-21ec59d95017        1 GiB  304 MiB

/ControllerReclaimSpace - sparsify command succeed

I0529 07:29:28.589924       1 utils.go:198] ID: 26 GRPC call: /reclaimspace.ReclaimSpaceController/ControllerReclaimSpace
I0529 07:29:28.595724       1 utils.go:199] ID: 26 GRPC request: {"parameters":{"clusterID":"rook-ceph","encryptionKMSID":"azure-test","imageFeatures":"layering","imageFormat":"2","imageName":"csi-vol-58a49f13-ff2e-4415-b1a8-21ec59d95017","journalPool":"replicapool","pool":"replicapool","storage.kubernetes.io/csiProvisionerIdentity":"1716966822191-8666-rook-ceph.rbd.csi.ceph.com"},"secrets":"***stripped***","volume_id":"0001-0009-rook-ceph-0000000000000008-58a49f13-ff2e-4415-b1a8-21ec59d95017"}
I0529 07:29:28.690999       1 omap.go:89] ID: 26 got omap values: (pool="replicapool", namespace="", name="csi.volume.58a49f13-ff2e-4415-b1a8-21ec59d95017"): map[csi.imageid:73c2aff800e79 csi.imagename:csi-vol-58a49f13-ff2e-4415-b1a8-21ec59d95017 csi.volname:pvc-ffd7ee84-6ae9-47fc-8eb0-32d9c255b92b csi.volume.owner:reclaim-ns]
I0529 07:29:28.761741       1 reclaimspace.go:77] ID: 26 volume with ID "0001-0009-rook-ceph-0000000000000008-58a49f13-ff2e-4415-b1a8-21ec59d95017" is in use, skipping sparsify operation
I0529 07:29:28.762193       1 utils.go:205] ID: 26 GRPC response: {}
I0529 07:31:26.161922       1 utils.go:198] ID: 27 GRPC call: /reclaimspace.ReclaimSpaceController/ControllerReclaimSpace
I0529 07:31:26.163979       1 utils.go:199] ID: 27 GRPC request: {"parameters":{"clusterID":"rook-ceph","encryptionKMSID":"azure-test","imageFeatures":"layering","imageFormat":"2","imageName":"csi-vol-58a49f13-ff2e-4415-b1a8-21ec59d95017","journalPool":"replicapool","pool":"replicapool","storage.kubernetes.io/csiProvisionerIdentity":"1716966822191-8666-rook-ceph.rbd.csi.ceph.com"},"secrets":"***stripped***","volume_id":"0001-0009-rook-ceph-0000000000000008-58a49f13-ff2e-4415-b1a8-21ec59d95017"}
I0529 07:31:26.171744       1 omap.go:89] ID: 27 got omap values: (pool="replicapool", namespace="", name="csi.volume.58a49f13-ff2e-4415-b1a8-21ec59d95017"): map[csi.imageid:73c2aff800e79 csi.imagename:csi-vol-58a49f13-ff2e-4415-b1a8-21ec59d95017 csi.volname:pvc-ffd7ee84-6ae9-47fc-8eb0-32d9c255b92b csi.volume.owner:reclaim-ns]
I0529 07:31:26.697576       1 utils.go:205] ID: 27 GRPC response: {}

[root@c1 /]# rbd du -p replicapool
warning: fast-diff map is not enabled for csi-vol-58a49f13-ff2e-4415-b1a8-21ec59d95017. operation may be slow.
NAME                                          PROVISIONED  USED
csi-vol-58a49f13-ff2e-4415-b1a8-21ec59d95017        1 GiB  28 MiB

@Madhu-1, will this be enough for the test?

[iPraveenParihar]

/test ci/centos/k8s-e2e-external-storage/1.28

[Madhu-1]

/test ci/centos/k8s-e2e-external-storage/1.29

[Madhu-1]

/test ci/centos/k8s-e2e-external-storage/1.28

[Madhu-1]

/test ci/centos/k8s-e2e-external-storage/1.28

[nixpanic]

/test ci/centos/k8s-e2e-external-storage/1.29

[nixpanic]

/test ci/centos/k8s-e2e-external-storage/1.30

[nixpanic]

/test ci/centos/k8s-e2e-external-storage/1.30

[ceph-csi-bot]

/test ci/centos/k8s-e2e-external-storage/1.27

[ceph-csi-bot]

/test ci/centos/upgrade-tests-cephfs

[ceph-csi-bot]

/test ci/centos/mini-e2e-helm/k8s-1.27

[ceph-csi-bot]

/test ci/centos/k8s-e2e-external-storage/1.28

[ceph-csi-bot]

/test ci/centos/k8s-e2e-external-storage/1.29

[ceph-csi-bot]

/test ci/centos/mini-e2e/k8s-1.27

[ceph-csi-bot]

/test ci/centos/upgrade-tests-rbd

[ceph-csi-bot]

/test ci/centos/mini-e2e-helm/k8s-1.28

[ceph-csi-bot]

/test ci/centos/mini-e2e-helm/k8s-1.29

[ceph-csi-bot]

/test ci/centos/k8s-e2e-external-storage/1.30

[ceph-csi-bot]

/test ci/centos/mini-e2e/k8s-1.28

[ceph-csi-bot]

/test ci/centos/mini-e2e/k8s-1.29

[ceph-csi-bot]

/test ci/centos/mini-e2e-helm/k8s-1.30

[ceph-csi-bot]

/test ci/centos/mini-e2e/k8s-1.30

[nixpanic]

/retest ci/centos/k8s-e2e-external-storage/1.27

[nixpanic]

/retest ci/centos/upgrade-tests-cephfs

[Madhu-1]

/retest ci/centos/upgrade-tests-cephfs

[nixpanic]

/test ci/centos/k8s-e2e-external-storage/1.30

[nixpanic]

/test ci/centos/k8s-e2e-external-storage/1.30

[ceph-csi-bot]

/test ci/centos/k8s-e2e-external-storage/1.27

[ceph-csi-bot]

/test ci/centos/k8s-e2e-external-storage/1.29

[ceph-csi-bot]

/test ci/centos/k8s-e2e-external-storage/1.30

[ceph-csi-bot]

/test ci/centos/k8s-e2e-external-storage/1.28

[ceph-csi-bot]

/test ci/centos/mini-e2e-helm/k8s-1.27

[ceph-csi-bot]

/test ci/centos/upgrade-tests-cephfs

[ceph-csi-bot]

/test ci/centos/mini-e2e-helm/k8s-1.29

[ceph-csi-bot]

/test ci/centos/mini-e2e/k8s-1.27

[ceph-csi-bot]

/test ci/centos/mini-e2e-helm/k8s-1.28

[ceph-csi-bot]

/test ci/centos/mini-e2e/k8s-1.29

[ceph-csi-bot]

/test ci/centos/upgrade-tests-rbd

[ceph-csi-bot]

/test ci/centos/mini-e2e-helm/k8s-1.30

[ceph-csi-bot]

/test ci/centos/mini-e2e/k8s-1.28

[ceph-csi-bot]

/test ci/centos/mini-e2e/k8s-1.30

[iPraveenParihar]

@Mergifyio rebase

[mergify[bot]]

rebase

✅ Branch has been successfully rebased