search

csi-addons / kubernetes-csi-addons

CSI-Addons implementation and APIs for Kubernetes
Apache License 2.0
16 stars 34 forks source link

reclaimspace: enhancement to consider existing PVCs for StorageClass Annotations #614

Closed iPraveenParihar closed 16 hours ago

iPraveenParihar commented 2 weeks ago

Currently, If annotations are set on StorageClass this will only affect new PVCs created from this StorageClass for ReclaimSpace operations. To include existing PVCs, one needs to restart the controller.

This commit eliminates the need of restarting the controller by watching the StorageClass object and Equeue all PVCs of it, this will then allow considering the existing PVCs and annotations will be set.

Initial testing -

RBD StorageClass

$ k get sc rook-ceph-block -oyaml
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  creationTimestamp: "2024-07-08T06:48:23Z"
  name: rook-ceph-block
  resourceVersion: "1879671"
  uid: 8e8e36e4-f0fb-460c-b352-3efd8799479e
parameters:
  clusterID: rook-ceph
  csi.storage.k8s.io/controller-expand-secret-name: rook-csi-rbd-provisioner
  csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph
  csi.storage.k8s.io/fstype: ext4
  csi.storage.k8s.io/node-stage-secret-name: rook-csi-rbd-node
  csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph
  csi.storage.k8s.io/provisioner-secret-name: rook-csi-rbd-provisioner
  csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph
  encrypted: "true"
  encryptionKMSID: azure-test
  imageFeatures: layering
  imageFormat: "2"
  pool: replicapool
provisioner: rook-ceph.rbd.csi.ceph.com
reclaimPolicy: Delete
volumeBindingMode: Immediate

RBD PVC

$ k get pvc rbd-pvc -oyaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: rook-ceph.rbd.csi.ceph.com
    volume.kubernetes.io/storage-provisioner: rook-ceph.rbd.csi.ceph.com
  creationTimestamp: "2024-07-11T07:09:18Z"
  finalizers:
  - kubernetes.io/pvc-protection
  name: rbd-pvc
  namespace: test
  resourceVersion: "2311226"
  uid: 7388ae15-3ee4-4051-90d5-0661181b36d6
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: rook-ceph-block
  volumeMode: Block
  volumeName: pvc-7388ae15-3ee4-4051-90d5-0661181b36d6
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 1Gi
  phase: Bound

Annotate the StorageClass and see that the PVCs created before are annotated

$ k annotate sc rook-ceph-block "reclaimspace.csiaddons.openshift.io/schedule=@daily"
storageclass.storage.k8s.io/rook-ceph-block annotated

$ k get pvc rbd-pvc -oyaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    reclaimspace.csiaddons.openshift.io/cronjob: rbd-pvc-1720682724
    reclaimspace.csiaddons.openshift.io/schedule: '@daily'
    volume.beta.kubernetes.io/storage-provisioner: rook-ceph.rbd.csi.ceph.com
    volume.kubernetes.io/storage-provisioner: rook-ceph.rbd.csi.ceph.com
  creationTimestamp: "2024-07-11T07:09:18Z"
  finalizers:
  - kubernetes.io/pvc-protection
  name: rbd-pvc
  namespace: test
  resourceVersion: "2312547"
  uid: 7388ae15-3ee4-4051-90d5-0661181b36d6
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: rook-ceph-block
  volumeMode: Block
  volumeName: pvc-7388ae15-3ee4-4051-90d5-0661181b36d6
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 1Gi
  phase: Bound

manager logs

2024-07-11T07:11:04.520Z    INFO    Annotation not set, exiting reconcile   {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"test"}, "namespace": "test", "name": "rbd-pvc", "reconcileID": "8521eaa0-3fc1-4541-8214-c66d34003ecf"}
2024-07-11T07:25:24.966Z    INFO    Adding annotation   {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"test"}, "namespace": "test", "name": "rbd-pvc", "reconcileID": "477395f4-160d-4d0b-b67d-033d69aa20a9", "Schedule": "@daily", "ReclaimSpaceCronJobName": "rbd-pvc-1720682724", "Annotation": "{\"metadata\":{\"annotations\":{\"reclaimspace.csiaddons.openshift.io/cronjob\":\"rbd-pvc-1720682724\",\"reclaimspace.csiaddons.openshift.io/schedule\":\"@daily\"}}}"}
2024-07-11T07:25:24.996Z    INFO    KubeAPIWarningLogger    unknown field "spec.jobTemplate.metadata.creationTimestamp"
2024-07-11T07:25:24.997Z    INFO    Successfully created reclaimSpaceCronJob    {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"test"}, "namespace": "test", "name": "rbd-pvc", "reconcileID": "477395f4-160d-4d0b-b67d-033d69aa20a9", "Schedule": "@daily", "ReclaimSpaceCronJobName": "rbd-pvc-1720682724"}
2024-07-11T07:25:25.017Z    INFO    No upcoming scheduled times, requeue with delay till next run   {"controller": "reclaimspacecronjob", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "ReclaimSpaceCronJob", "ReclaimSpaceCronJob": {"name":"rbd-pvc-1720682724","namespace":"test"}, "namespace": "test", "name": "rbd-pvc-1720682724", "reconcileID": "7761b968-0fda-4e2b-8abd-4bde8f3c3854", "now": "2024-07-11T07:25:25.017Z", "nextRun": "2024-07-12T00:00:00.000Z"}
2024-07-11T07:25:25.019Z    ERROR   Failed to update reclaimSpaceCronJob    {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"test"}, "namespace": "test", "name": "rbd-pvc", "reconcileID": "4ca1a3ee-7162-434e-9694-7295c5e3b939", "ReclaimSpaceCronJobName": "rbd-pvc-1720682724", "Schedule": "@daily", "error": "Operation cannot be fulfilled on reclaimspacecronjobs.csiaddons.openshift.io \"rbd-pvc-1720682724\": the object has been modified; please apply your changes to the latest version and try again"}
2024-07-11T07:25:25.021Z    ERROR   Reconciler error    {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"test"}, "namespace": "test", "name": "rbd-pvc", "reconcileID": "4ca1a3ee-7162-434e-9694-7295c5e3b939", "error": "Operation cannot be fulfilled on reclaimspacecronjobs.csiaddons.openshift.io \"rbd-pvc-1720682724\": the object has been modified;
2024-07-11T07:25:25.030Z    INFO    No upcoming scheduled times, requeue with delay till next run   {"controller": "reclaimspacecronjob", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "ReclaimSpaceCronJob", "ReclaimSpaceCronJob": {"name":"rbd-pvc-1720682724","namespace":"test"}, "namespace": "test", "name": "rbd-pvc-1720682724", "reconcileID": "dd49f11d-a0d5-4349-9417-f0185933a84b", "now": "2024-07-11T07:25:25.030Z", "nextRun": "2024-07-12T00:00:00.000Z"}
2024-07-11T07:25:25.037Z    INFO    Successfully updated reclaimSpaceCronJob    {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc","namespace":"test"}, "namespace": "test", "name": "rbd-pvc", "reconcileID": "654882a0-7e69-4aa3-84b2-d5ab3668466e", "ReclaimSpaceCronJobName": "rbd-pvc-1720682724", "Schedule": "@daily"}

Created new RBD PVC and annotations are set

$ k create -f pvc.yaml
persistentvolumeclaim/rbd-pvc-new created

$ k get pvc rbd-pvc-new -oyaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    reclaimspace.csiaddons.openshift.io/cronjob: rbd-pvc-new-1720683030
    reclaimspace.csiaddons.openshift.io/schedule: '@daily'
    volume.beta.kubernetes.io/storage-provisioner: rook-ceph.rbd.csi.ceph.com
    volume.kubernetes.io/storage-provisioner: rook-ceph.rbd.csi.ceph.com
  creationTimestamp: "2024-07-11T07:30:24Z"
  finalizers:
  - kubernetes.io/pvc-protection
  name: rbd-pvc-new
  namespace: test
  resourceVersion: "2313046"
  uid: decb16c6-44be-45d2-ba75-7b4ab029be9d
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: rook-ceph-block
  volumeMode: Block
  volumeName: pvc-decb16c6-44be-45d2-ba75-7b4ab029be9d
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 1Gi
  phase: Bound

manager logs

2024-07-11T07:30:27.566Z    INFO    PVC is not in bound state   {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc-new","namespace":"test"}, "namespace": "test", "name": "rbd-pvc-new", "reconcileID": "1d67ad16-003a-434a-a297-82877fa44f8d", "PVCPhase": "Pending"}
2024-07-11T07:30:30.128Z    INFO    Adding annotation   {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc-new","namespace":"test"}, "namespace": "test", "name": "rbd-pvc-new", "reconcileID": "43bae20b-8561-4e55-8769-9f8708fead39", "Schedule": "@daily", "ReclaimSpaceCronJobName": "rbd-pvc-new-1720683030", "Annotation": "{\"metadata\":{\"annotations\":{\"reclaimspace.csiaddons.openshift.io/cronjob\":\"rbd-pvc-new-1720683030\",\"reclaimspace.csiaddons.openshift.io/schedule\":\"@daily\"}}}"}
2024-07-11T07:30:30.150Z    INFO    Successfully created reclaimSpaceCronJob    {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc-new","namespace":"test"}, "namespace": "test", "name": "rbd-pvc-new", "reconcileID": "43bae20b-8561-4e55-8769-9f8708fead39", "Schedule": "@daily", "ReclaimSpaceCronJobName": "rbd-pvc-new-1720683030"}
2024-07-11T07:30:30.159Z    INFO    Successfully updated reclaimSpaceCronJob    {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc-new","namespace":"test"}, "namespace": "test", "name": "rbd-pvc-new", "reconcileID": "3fd51186-854a-445e-8a8c-88667e33e479", "ReclaimSpaceCronJobName": "rbd-pvc-new-1720683030", "Schedule": "@daily"}
2024-07-11T07:30:30.165Z    INFO    No upcoming scheduled times, requeue with delay till next run   {"controller": "reclaimspacecronjob", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "ReclaimSpaceCronJob", "ReclaimSpaceCronJob": {"name":"rbd-pvc-new-1720683030","namespace":"test"}, "namespace": "test", "name": "rbd-pvc-new-1720683030", "reconcileID": "e9c95d30-2f78-4625-a3b6-bf4c3c861c76", "now": "2024-07-11T07:30:30.165Z", "nextRun": "2024-07-12T00:00:00.000Z"}
2024-07-11T07:30:30.170Z    ERROR   Failed to update reclaimSpaceCronJob    {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc-new","namespace":"test"}, "namespace": "test", "name": "rbd-pvc-new", "reconcileID": "0b54e3b8-1c01-49dd-b571-ee7935c8c3b6", "ReclaimSpaceCronJobName": "rbd-pvc-new-1720683030", "Schedule": "@daily", "error": "Operation cannot be fulfilled on reclaimspacecronjobs.csiaddons.openshift.io \"rbd-pvc-new-1720683030\": the object has been
2024-07-11T07:30:30.170Z    ERROR   Reconciler error    {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc-new","namespace":"test"}, "namespace": "test", "name": "rbd-pvc-new", "reconcileID": "0b54e3b8-1c01-49dd-b571-ee7935c8c3b6", "error": "Operation cannot be fulfilled on reclaimspacecronjobs.csiaddons.openshift.io \"rbd-pvc-new-1720683030\": the object has been modified; please apply your changes to the latest version and try again"}
2024-07-11T07:30:30.176Z    INFO    No upcoming scheduled times, requeue with delay till next run   {"controller": "reclaimspacecronjob", "controllerGroup": "csiaddons.openshift.io", "controllerKind": "ReclaimSpaceCronJob", "ReclaimSpaceCronJob": {"name":"rbd-pvc-new-1720683030","namespace":"test"}, "namespace": "test", "name": "rbd-pvc-new-1720683030", "reconcileID": "d269ebe5-1b02-4aba-b9c8-27d6655f3fc3", "now": "2024-07-11T07:30:30.175Z", "nextRun": "2024-07-12T00:00:00.000Z"}
2024-07-11T07:30:30.184Z    INFO    Successfully updated reclaimSpaceCronJob    {"controller": "persistentvolumeclaim", "controllerGroup": "", "controllerKind": "PersistentVolumeClaim", "PersistentVolumeClaim": {"name":"rbd-pvc-new","namespace":"test"}, "namespace": "test", "name": "rbd-pvc-new", "reconcileID": "480a067e-56d5-4d9f-94db-7e8d6ea94706", "ReclaimSpaceCronJobName": "rbd-pvc-new-1720683030", "Schedule": "@daily"}
Madhu-1 commented 4 days ago

@iPraveenParihar anything pending on this one?

iPraveenParihar commented 4 days ago

@iPraveenParihar anything pending on this one?

This PR covers the Reconciling of existing PVCs(with no ReclaimSpace annotation) when ReclaimSpace annotation is added on SC.

it looks like we will skip these pvcs, won't this cause a problem when an annotation is deleted on the storageclass and the controller-added annotation exists on the PVC, IMO we need an owner to identify who added the annotation is it by the controller or not.

Right now am working on the above scenario - I think we can merge this PR? and I'll raise different PR to cover the above case?

Madhu-1 commented 4 days ago

@nixpanic @Rakshith-R PTAL

iPraveenParihar commented 18 hours ago

resolved conflicts