search

kubernetes-csi / external-snapshotter

Sidecar container that watches Kubernetes Snapshot CRD objects and triggers CreateSnapshot/DeleteSnapshot against a CSI endpoint.
Apache License 2.0
501 stars 374 forks source link

volumesnapshot and volumesnapshotcontent not getting created #1191

Closed yati1998 closed 5 days ago

yati1998 commented 2 weeks ago

With the current master branch, while dynamically creating volumegroupsnapshot, the volumesnapshot and volumesnapshot content is not getting created. There seems to be some breaking change which is merged in master.

yati@dhcp53-199:~/csi-driver-host-path$ kubectl get pvc csi-pvc -oyaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: hostpath.csi.k8s.io
    volume.kubernetes.io/storage-provisioner: hostpath.csi.k8s.io
  creationTimestamp: "2024-11-08T04:01:12Z"
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    group: test
  name: csi-pvc
  namespace: default
  resourceVersion: "1932"
  uid: 4017dabc-a4d2-40b4-b6f7-8da3879fb909
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: csi-hostpath-sc
  volumeMode: Filesystem
  volumeName: pvc-4017dabc-a4d2-40b4-b6f7-8da3879fb909
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 1Gi
  phase: Bound
yati@dhcp53-199:~/csi-driver-host-path$ kubectl get volumegroupsnapshot
NAME                       READYTOUSE   VOLUMEGROUPSNAPSHOTCLASS      VOLUMEGROUPSNAPSHOTCONTENT                              CREATIONTIME   AGE
new-groupsnapshot-master   true         csi-hostpath-groupsnapclass   groupsnapcontent-c7e01770-7cab-4fa4-881c-e286d7c16fb0   75s            75s
yati@dhcp53-199:~/csi-driver-host-path$ kubectl get volumegroupsnapshot new-groupsnapshot-master -oyaml
apiVersion: groupsnapshot.storage.k8s.io/v1alpha1
kind: VolumeGroupSnapshot
metadata:
  creationTimestamp: "2024-11-08T04:03:04Z"
  finalizers:
  - groupsnapshot.storage.kubernetes.io/volumegroupsnapshot-bound-protection
  generation: 1
  name: new-groupsnapshot-master
  namespace: default
  resourceVersion: "2005"
  uid: c7e01770-7cab-4fa4-881c-e286d7c16fb0
spec:
  source:
    selector:
      matchLabels:
        group: test
  volumeGroupSnapshotClassName: csi-hostpath-groupsnapclass
status:
  boundVolumeGroupSnapshotContentName: groupsnapcontent-c7e01770-7cab-4fa4-881c-e286d7c16fb0
  creationTime: "2024-11-08T04:03:04Z"
  readyToUse: true
yati@dhcp53-199:~/csi-driver-host-path$ kubectl get volumesnapshot
No resources found in default namespace.
yati@dhcp53-199:~/csi-driver-host-path$ kubectl get volumesnapshotcontent
No resources found
yati1998 commented 2 weeks ago

even the deletion is getting stuck.

yati1998 commented 2 weeks ago

it is workin fine with v8.1.0 though:

yati@dhcp53-199:~/csi-driver-host-path$ kubectl get volumesnapshot
NAME                                                                                           READYTOUSE   SOURCEPVC   SOURCESNAPSHOTCONTENT                                                                             RESTORESIZE   SNAPSHOTCLASS   SNAPSHOTCONTENT                                                                                   CREATIONTIME   AGE
snapshot-7897054076dce494b65983a74f66f0f6dd60653cd647bc68d1844105a71c62a8-2024-11-08-4.19.16   true                     snapcontent-7897054076dce494b65983a74f66f0f6dd60653cd647bc68d1844105a71c62a8-2024-11-08-4.19.16   1Gi                           snapcontent-7897054076dce494b65983a74f66f0f6dd60653cd647bc68d1844105a71c62a8-2024-11-08-4.19.16   6s             6s
yati@dhcp53-199:~/csi-driver-host-path$ kubectl get volumesnapshotcontent
NAME                                                                                              READYTOUSE   RESTORESIZE   DELETIONPOLICY   DRIVER                VOLUMESNAPSHOTCLASS   VOLUMESNAPSHOT                                                                                 VOLUMESNAPSHOTNAMESPACE   AGE
snapcontent-7897054076dce494b65983a74f66f0f6dd60653cd647bc68d1844105a71c62a8-2024-11-08-4.19.16   true         1073741824    Delete           hostpath.csi.k8s.io                         snapshot-7897054076dce494b65983a74f66f0f6dd60653cd647bc68d1844105a71c62a8-2024-11-08-4.19.16   default                   17s
yati1998 commented 1 week ago

@leonardoce your help on debugging this would be really appreciated. Let me know if you can help me on this?

leonardoce commented 1 week ago

Can you share the log of the snapshot controller? Did you apply the latest RBAC rules?

leonardoce commented 1 week ago

I'on commit 2965ff71c78a6d641496572fe9340b2c21d398c2 and:

apiVersion: groupsnapshot.storage.k8s.io/v1beta1
kind: VolumeGroupSnapshot
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"groupsnapshot.storage.k8s.io/v1alpha1","kind":"VolumeGroupSnapshot","metadata":{"annotations":{},"name":"new-groupsnapshot-demo","namespace":"default"},"spec":{"source":{"selector":{"matchLabels":{"cnpg.io/instanceName":"cluster-example-1"}}},"volumeGroupSnapshotClassName":"csi-hostpath-groupsnapclass"}}
  creationTimestamp: "2024-11-11T07:51:57Z"
  finalizers:
  - groupsnapshot.storage.kubernetes.io/volumegroupsnapshot-bound-protection
  generation: 1
  name: new-groupsnapshot-demo
  namespace: default
  resourceVersion: "559224"
  uid: 422d3cf1-d419-428c-babf-ea4a78e59280
spec:
  source:
    selector:
      matchLabels:
        cnpg.io/instanceName: cluster-example-1
  volumeGroupSnapshotClassName: csi-hostpath-groupsnapclass
status:
  boundVolumeGroupSnapshotContentName: groupsnapcontent-422d3cf1-d419-428c-babf-ea4a78e59280
  creationTime: "2024-11-11T07:51:57Z"
  pvcVolumeSnapshotRefList:
  - persistentVolumeClaimRef:
      name: cluster-example-1
    volumeSnapshotRef:
      name: snapshot-c099a6b2c601b5efd09f4f03f4ae7bebcebb46c9f35836a30e7e391dd004375f
  - persistentVolumeClaimRef:
      name: cluster-example-1-wal
    volumeSnapshotRef:
      name: snapshot-43e9b6f832ea5881ff54a338fee5ef0e29b02cb8d9aa12e411dfd33e21b9aa9a
  readyToUse: true
---
apiVersion: groupsnapshot.storage.k8s.io/v1beta1
kind: VolumeGroupSnapshotContent
metadata:
  creationTimestamp: "2024-11-11T07:51:57Z"
  finalizers:
  - groupsnapshot.storage.kubernetes.io/volumegroupsnapshotcontent-bound-protection
  generation: 1
  name: groupsnapcontent-422d3cf1-d419-428c-babf-ea4a78e59280
  resourceVersion: "559215"
  uid: 6fb2f09d-20a1-4b21-abcb-b0a4fe438440
spec:
  deletionPolicy: Delete
  driver: hostpath.csi.k8s.io
  source:
    volumeHandles:
    - 90080ce0-9db8-11ef-92da-8268f90a9699
    - 9008d486-9db8-11ef-92da-8268f90a9699
  volumeGroupSnapshotClassName: csi-hostpath-groupsnapclass
  volumeGroupSnapshotRef:
    apiVersion: groupsnapshot.storage.k8s.io/v1alpha1
    kind: VolumeGroupSnapshot
    name: new-groupsnapshot-demo
    namespace: default
    resourceVersion: "559184"
    uid: 422d3cf1-d419-428c-babf-ea4a78e59280
status:
  creationTime: 1731311517285893289
  pvVolumeSnapshotContentList:
  - persistentVolumeRef:
      name: pvc-a02df2b4-3aae-4916-b781-8d1f9cedbad4
    volumeSnapshotContentRef:
      name: snapcontent-c099a6b2c601b5efd09f4f03f4ae7bebcebb46c9f35836a30e7e391dd004375f
  - persistentVolumeRef:
      name: pvc-b4219908-e9de-4c8e-81ee-65973b05c798
    volumeSnapshotContentRef:
      name: snapcontent-43e9b6f832ea5881ff54a338fee5ef0e29b02cb8d9aa12e411dfd33e21b9aa9a
  readyToUse: true
  volumeGroupSnapshotHandle: d356a8c8-a001-11ef-9e36-c64721e32878
  volumeSnapshotHandlePairList:
  - snapshotHandle: d356ac43-a001-11ef-9e36-c64721e32878
    volumeHandle: 90080ce0-9db8-11ef-92da-8268f90a9699
  - snapshotHandle: d3cb10a8-a001-11ef-9e36-c64721e32878
    volumeHandle: 9008d486-9db8-11ef-92da-8268f90a9699

This are my volumesnapshots:

apiVersion: v1
items:
- apiVersion: snapshot.storage.k8s.io/v1
  kind: VolumeSnapshot
  metadata:
    creationTimestamp: "2024-11-11T07:51:59Z"
    finalizers:
    - snapshot.storage.kubernetes.io/volumesnapshot-in-group-protection
    - snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection
    generation: 1
    labels:
      groupsnapshot.storage.k8s.io/volumeGroupSnapshotName: new-groupsnapshot-demo
    name: snapshot-43e9b6f832ea5881ff54a338fee5ef0e29b02cb8d9aa12e411dfd33e21b9aa9a
    namespace: default
    resourceVersion: "559221"
    uid: 8439217c-c238-4115-8bb6-cd5dfe9b0aa9
  spec:
    source:
      persistentVolumeClaimName: cluster-example-1-wal
  status:
    boundVolumeSnapshotContentName: snapcontent-43e9b6f832ea5881ff54a338fee5ef0e29b02cb8d9aa12e411dfd33e21b9aa9a
    creationTime: "2024-11-11T07:51:57Z"
    readyToUse: true
    restoreSize: 1Gi
    volumeGroupSnapshotName: new-groupsnapshot-demo
- apiVersion: snapshot.storage.k8s.io/v1
  kind: VolumeSnapshot
  metadata:
    creationTimestamp: "2024-11-11T07:51:59Z"
    finalizers:
    - snapshot.storage.kubernetes.io/volumesnapshot-in-group-protection
    - snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection
    generation: 1
    labels:
      groupsnapshot.storage.k8s.io/volumeGroupSnapshotName: new-groupsnapshot-demo
    name: snapshot-c099a6b2c601b5efd09f4f03f4ae7bebcebb46c9f35836a30e7e391dd004375f
    namespace: default
    resourceVersion: "559209"
    uid: e8ed7894-d9f8-48c3-8788-1970bbcc5f67
  spec:
    source:
      persistentVolumeClaimName: cluster-example-1
  status:
    boundVolumeSnapshotContentName: snapcontent-c099a6b2c601b5efd09f4f03f4ae7bebcebb46c9f35836a30e7e391dd004375f
    creationTime: "2024-11-11T07:51:57Z"
    readyToUse: true
    restoreSize: 1Gi
    volumeGroupSnapshotName: new-groupsnapshot-demo
kind: List
metadata:
  resourceVersion: ""

and their contents:

apiVersion: v1
items:
- apiVersion: snapshot.storage.k8s.io/v1
  kind: VolumeSnapshotContent
  metadata:
    creationTimestamp: "2024-11-11T07:51:59Z"
    finalizers:
    - snapshot.storage.kubernetes.io/volumesnapshotcontent-bound-protection
    generation: 2
    name: snapcontent-43e9b6f832ea5881ff54a338fee5ef0e29b02cb8d9aa12e411dfd33e21b9aa9a
    resourceVersion: "559213"
    uid: 4d52976c-4e2e-436f-bc78-5eb7a24a96f1
  spec:
    deletionPolicy: Delete
    driver: hostpath.csi.k8s.io
    source:
      snapshotHandle: d3cb10a8-a001-11ef-9e36-c64721e32878
    sourceVolumeMode: Filesystem
    volumeSnapshotRef:
      kind: VolumeSnapshot
      name: snapshot-43e9b6f832ea5881ff54a338fee5ef0e29b02cb8d9aa12e411dfd33e21b9aa9a
      namespace: default
      uid: 8439217c-c238-4115-8bb6-cd5dfe9b0aa9
  status:
    creationTime: 1731311517285893289
    readyToUse: true
    restoreSize: 1073741824
    snapshotHandle: d3cb10a8-a001-11ef-9e36-c64721e32878
    volumeGroupSnapshotHandle: d356a8c8-a001-11ef-9e36-c64721e32878
- apiVersion: snapshot.storage.k8s.io/v1
  kind: VolumeSnapshotContent
  metadata:
    creationTimestamp: "2024-11-11T07:51:59Z"
    finalizers:
    - snapshot.storage.kubernetes.io/volumesnapshotcontent-bound-protection
    generation: 2
    name: snapcontent-c099a6b2c601b5efd09f4f03f4ae7bebcebb46c9f35836a30e7e391dd004375f
    resourceVersion: "559203"
    uid: cd5685f8-d9ad-4015-a9ed-2409b716c5b1
  spec:
    deletionPolicy: Delete
    driver: hostpath.csi.k8s.io
    source:
      snapshotHandle: d356ac43-a001-11ef-9e36-c64721e32878
    sourceVolumeMode: Filesystem
    volumeSnapshotRef:
      kind: VolumeSnapshot
      name: snapshot-c099a6b2c601b5efd09f4f03f4ae7bebcebb46c9f35836a30e7e391dd004375f
      namespace: default
      uid: e8ed7894-d9f8-48c3-8788-1970bbcc5f67
  status:
    creationTime: 1731311517285893289
    readyToUse: true
    restoreSize: 1073741824
    snapshotHandle: d356ac43-a001-11ef-9e36-c64721e32878
    volumeGroupSnapshotHandle: d356a8c8-a001-11ef-9e36-c64721e32878
kind: List
metadata:
  resourceVersion: ""

Deletion won't work in your case because the finalizer is preventing you deleting them until they're correctly created.