search

ceph / ceph-csi

CSI driver for Ceph
Apache License 2.0
1.29k stars 551 forks source link

velero ceph-csi snapshot error external-ceph kubernetes #3912

Closed Borosan closed 1 year ago

Borosan commented 1 year ago

Describe the bug

unable to get snapshot from ceph-csi provisioned pvc on kubernetes (external ceph) using volumesnapshot or velero

Environment details

-velero version (you can forget about velero , it was just my starting point, i cant event take a manual volume snap shot) Client: Version: v1.10.3 Git commit: 80cc81b540a738869546890a43255c5f9ea454ae Server: Version: v1.11.0

Steps to reproduce

install a kubernetes cluster using kubeadm (3 master-3worker)

install a ceph cluster using cephadm (1 cephadm, 6 nodes(3mon,3osd)) connect your kubernetes cluster to ceph cluster using ceph-csi-rbd url: url: https://github.com/kubernetes-csi/external-snapshotter/tree/v5.0.0

proviosn some pvs and pvc

i have used velero cli example nginx app with pv: url:https://github.com/vmware-tanzu/velero/releases/download/v1.10.3/velero-v1.10.3-linux-amd64.tar.gz use nginx app with.pv.yaml edit storage class to what your ceph csi rbd is

deploy kubernetes exsternal snapshotter v5.0.0 (5.0.1and 6.0.0 and above also tested) i have also tried (url:https://github.com/ceph/ceph-csi/blob/devel/docs/snap-clone.md)

git clone https://github.com/kubernetes-csi/external-snapshotter.git --branch v5.0.0 --depth 1
cd external-snapshotter
kubectl -n kube-system create -k client/config/crd/
kubectl -n kube-system create -k deploy/kubernetes/snapshot-controller

deploy ceph-csi-rbd snapshotclass:

apiVersion: snapshot.storage.k8s.io/v1 kind: VolumeSnapshotClass metadata: name: csi-rbdplugin-snapclass labels: velero.io/csi-volumesnapshot-class: "true" driver: rbd.csi.ceph.com parameters: csi.storage.k8s.io/snapshotter-secret-name: csi-rbd-secret csi.storage.k8s.io/snapshotter-secret-namespace: ceph-csi-rbd deletionPolicy: Delete *(my additional note: i have tested v1beta1 too! same problems, i have changed namespace to default but not resolved)

get volumesnapshot from your sample app. (manually)

apiVersion: snapshot.storage.k8s.io/v1beta1 kind: VolumeSnapshot metadata: name: rbd-pvc-snapshot spec: volumeSnapshotClassName: csi-rbdplugin-snapclass source: persistentVolumeClaimName: rbd-pvc *(additional my note: i have tested from different namespaces)

Actual results

no successfull volumesnapshot taken

Expected behavior

have a healthy volumesnapshot

logs

taken volumesnapshot:

 k get volumesnapshot rbd-pvc-snapshot
NAME               READYTOUSE   SOURCEPVC   SOURCESNAPSHOTCONTENT   RESTORESIZE   SNAPSHOTCLASS             SNAPSHOTCONTENT   CREATIONTIME   AGE
rbd-pvc-snapshot   false        rbd-pvc                                           csi-rbdplugin-snapclass                                    41m

describe:

snapshot-controller  Failed to create snapshot content with error snapshot controller failed to update rbd-pvc-snapshot on API server: cannot get claim from snapshot

snapshot controller logs:

I0618 08:12:02.859255       1 snapshot_controller_base.go:227] syncSnapshotByKey[nginx-example/rbd-pvc-snapshot]
I0618 08:12:02.859287       1 snapshot_controller_base.go:230] snapshotWorker: snapshot namespace [nginx-example] name [rbd-pvc-snapshot]
I0618 08:12:02.859298       1 snapshot_controller_base.go:353] checkAndUpdateSnapshotClass [rbd-pvc-snapshot]: VolumeSnapshotClassName [csi-rbdplugin-snapclass]
I0618 08:12:02.859304       1 snapshot_controller.go:1284] getSnapshotClass: VolumeSnapshotClassName [csi-rbdplugin-snapclass]
I0618 08:12:02.859311       1 snapshot_controller_base.go:373] VolumeSnapshotClass [csi-rbdplugin-snapclass] Driver [rbd.csi.ceph.com]
I0618 08:12:02.859319       1 snapshot_controller_base.go:247] Updating snapshot "nginx-example/rbd-pvc-snapshot"
I0618 08:12:02.859326       1 snapshot_controller_base.go:383] updateSnapshot "nginx-example/rbd-pvc-snapshot"
I0618 08:12:02.859339       1 util.go:218] storeObjectUpdate updating snapshot "nginx-example/rbd-pvc-snapshot" with version 152781813
I0618 08:12:02.859357       1 snapshot_controller.go:183] synchronizing VolumeSnapshot[nginx-example/rbd-pvc-snapshot]: bound to: "", Completed: false
I0618 08:12:02.859388       1 snapshot_controller.go:185] syncSnapshot [nginx-example/rbd-pvc-snapshot]: check if we should remove finalizer on snapshot PVC source and remove it if we can
I0618 08:12:02.859405       1 snapshot_controller.go:978] cannot get claim from snapshot [rbd-pvc-snapshot]: [failed to retrieve PVC rbd-pvc from the lister: "persistentvolumeclaim \"rbd-pvc\" not found"] Claim may be deleted already. No need to remove finalizer on the claim.
I0618 08:12:02.859419       1 snapshot_controller.go:194] syncSnapshot[nginx-example/rbd-pvc-snapshot]: check if we should add invalid label on snapshot
I0618 08:12:02.859429       1 snapshot_controller.go:210] syncSnapshot[nginx-example/rbd-pvc-snapshot]: validate snapshot to make sure source has been correctly specified
I0618 08:12:02.859436       1 snapshot_controller.go:219] syncSnapshot[nginx-example/rbd-pvc-snapshot]: check if we should add finalizers on snapshot
I0618 08:12:02.859461       1 snapshot_controller.go:418] syncUnreadySnapshot nginx-example/rbd-pvc-snapshot
I0618 08:12:02.859465       1 snapshot_controller.go:1298] getSnapshotDriverName: VolumeSnapshot[rbd-pvc-snapshot]
I0618 08:12:02.859469       1 snapshot_controller.go:1284] getSnapshotClass: VolumeSnapshotClassName [csi-rbdplugin-snapclass]
I0618 08:12:02.859482       1 snapshot_controller.go:479] getDynamicallyProvisionedContentFromStore for snapshot nginx-example/rbd-pvc-snapshot
I0618 08:12:02.859489       1 snapshot_controller.go:636] createSnapshotContent: Creating content for snapshot nginx-example/rbd-pvc-snapshot through the plugin ...
I0618 08:12:02.859493       1 snapshot_controller.go:639] createSnapshotContent: Check if PVC is not being deleted and add Finalizer for source of snapshot [rbd-pvc-snapshot] if needed
I0618 08:12:02.859505       1 snapshot_controller.go:890] cannot get claim from snapshot [rbd-pvc-snapshot]: [failed to retrieve PVC rbd-pvc from the lister: "persistentvolumeclaim \"rbd-pvc\" not found"] Claim may be deleted already.
E0618 08:12:02.859519       1 snapshot_controller.go:642] createSnapshotContent failed to add finalizer for source of snapshot snapshot controller failed to update rbd-pvc-snapshot on API server: cannot get claim from snapshot
I0618 08:12:02.859543       1 snapshot_controller.go:782] updateSnapshotErrorStatusWithEvent[nginx-example/rbd-pvc-snapshot]
I0618 08:12:02.859573       1 snapshot_controller.go:785] updateSnapshotErrorStatusWithEvent[rbd-pvc-snapshot]: the same error &{2023-06-18 08:11:05 +0000 UTC 0xc0002f8960} is already set
E0618 08:12:02.859604       1 snapshot_controller_base.go:399] could not sync snapshot "nginx-example/rbd-pvc-snapshot": snapshot controller failed to update rbd-pvc-snapshot on API server: cannot get claim from snapshot
I0618 08:12:02.859652       1 snapshot_controller_base.go:217] Failed to sync snapshot "nginx-example/rbd-pvc-snapshot", will retry again: snapshot controller failed to update rbd-pvc-snapshot on API server: cannot get claim from snapshot
I0618 08:12:07.785588       1 leaderelection.go:278] successfully renewed lease kube-system/snapshot-controller-leader

velero logs:

nginx-logs-gk26j namespace=nginx-example resource=volumesnapshots.snapshot.storage.k8s.io
time="2023-06-18T08:57:57Z" level=info msg="Executing VolumeSnapshotBackupItemAction" backup=velero/nginx-backup cmd=/plugins/velero-plugin-for-csi logSource="/go/src/velero-plugin-for-csi/internal/backup/volumesnapshot_action.go:58" pluginName=velero-plugin-for-csi
time="2023-06-18T08:57:57Z" level=info msg="Getting VolumesnapshotContent for Volumesnapshot nginx-example/velero-nginx-logs-gk26j" backup=velero/nginx-backup cmd=/plugins/velero-plugin-for-csi logSource="/go/src/velero-plugin-for-csi/internal/backup/volumesnapshot_action.go:91" pluginName=velero-plugin-for-csi
time="2023-06-18T08:57:57Z" level=info msg="Waiting for CSI driver to reconcile volumesnapshot nginx-example/velero-nginx-logs-gk26j. Retrying in 5s" backup=velero/nginx-backup cmd=/plugins/velero-plugin-for-csi logSource="/go/src/velero-plugin-for-csi/internal/util/util.go:164" pluginName=velero-plugin-for-csi
time="2023-06-18T08:58:02Z" level=info msg="Waiting for volumesnapshotcontents snapcontent-c6c311f3-11f7-420a-8650-8fb62265e7b0 to have snapshot handle. Retrying in 5s" backup=velero/nginx-backup cmd=/plugins/velero-plugin-for-csi logSource="/go/src/velero-plugin-for-csi/internal/util/util.go:177" pluginName=velero-plugin-for-csi
time="2023-06-18T08:58:02Z" level=warning msg="Volumesnapshotcontent snapcontent-c6c311f3-11f7-420a-8650-8fb62265e7b0 has error: 0xc0007aeff0" backup=velero/nginx-backup cmd=/plugins/velero-plugin-for-csi logSource="/go/src/velero-plugin-for-csi/internal/util/util.go:179" pluginName=velero-plugin-for-csi
time="2023-06-18T08:58:07Z" level=info msg="Waiting for volumesnapshotcontents snapcontent-c6c311f3-11f7-420a-8650-8fb62265e7b0 to have snapshot handle. Retrying in 5s" backup=velero/nginx-backup cmd=/plugins/velero-plugin-for-csi logSource="/go/src/velero-plugin-for-csi/internal/util/util.go:177" pluginName=velero-plugin-for-csi
time="2023-06-18T08:58:07Z" level=warning msg="Volumesnapshotcontent snapcontent-c6c311f3-11f7-420a-8650-8fb62265e7b0 has error: 0xc000470820" backup=velero/nginx-backup cmd=/plugins/velero-plugin-for-csi logSource="/go/src/velero-plugin-for-csi/internal/util/util.go:179" pluginName=velero-plugin-for-csi
time="2023-06-18T08:58:12Z" level=info msg="Waiting for volumesnapshotcontents snapcontent-c6c311f3-11f7-420a-8650-8fb62265e7b0 to have snapshot handle. Retrying in 5s" backup=velero/nginx-backup cmd=/plugins/velero-plugin-for-csi logSource="/go/src/velero-plugin-for-csi/internal/util/util.go:177" pluginName=velero-plugin-for-csi
time="2023-06-18T08:58:12Z" level=warning msg="Volumesnapshotcontent snapcontent-c6c311f3-11f7-420a-8650-8fb62265e7b0 has error: 0xc000470d00" backup=velero/nginx-backup cmd=/plugins/velero-plugin-for-csi logSource="/go/src/velero-plugin-for-csi/internal/util/util.go:179" pluginName=velero-plugin-for-csi
.
.
.

Additional context

i have do the same thing with external-snapshotter v5.0.0, and longhorn 1.3.2 . i tried to find the right version between external snapshotter 5.0.x, 6.0.x and api v1beta1, v1 but no success,

can anybody tell me what is wrong? mismatch or .... if you have done this, tell me about method and versions.

Madhu-1 commented 1 year ago

@Borosan as suggested in the issue template, please provide the logs of the exteranal-snapshotter or csi plugin logs to check what is wrong.

Borosan commented 1 year ago

Okey, i redeployed every thing. and same resullt. here you are.

k logs  ceph-csi-rbd-nodeplugin-fzgjc
Defaulted container "driver-registrar" out of: driver-registrar, csi-rbdplugin, liveness-prometheus
I0528 11:08:41.527209    2565 main.go:166] Version: v2.6.2
I0528 11:08:41.527330    2565 main.go:167] Running node-driver-registrar in mode=registration
I0528 11:08:41.527847    2565 main.go:191] Attempting to open a gRPC connection with: "/csi/csi.sock"
I0528 11:08:41.538494    2565 connection.go:154] Connecting to unix:///csi/csi.sock
W0528 11:08:51.538668    2565 connection.go:173] Still connecting to unix:///csi/csi.sock
I0528 11:08:53.688685    2565 main.go:198] Calling CSI driver to discover driver name
I0528 11:08:53.688716    2565 connection.go:183] GRPC call: /csi.v1.Identity/GetPluginInfo
I0528 11:08:53.688721    2565 connection.go:184] GRPC request: {}
I0528 11:08:53.723361    2565 connection.go:186] GRPC response: {"name":"rbd.csi.ceph.com","vendor_version":"v3.8.0"}
I0528 11:08:53.723458    2565 connection.go:187] GRPC error: <nil>
I0528 11:08:53.723472    2565 main.go:208] CSI driver name: "rbd.csi.ceph.com"
I0528 11:08:53.733682    2565 node_register.go:53] Starting Registration Server at: /registration/rbd.csi.ceph.com-reg.sock
I0528 11:08:53.733952    2565 node_register.go:62] Registration Server started at: /registration/rbd.csi.ceph.com-reg.sock
I0528 11:08:53.734745    2565 node_register.go:92] Skipping HTTP server because endpoint is set to: ""
I0528 11:08:53.936908    2565 main.go:102] Received GetInfo call: &InfoRequest{}
I0528 11:08:53.937252    2565 main.go:109] "Kubelet registration probe created" path="/var/lib/kubelet/plugins/rbd.csi.ceph.com/registration"
I0528 11:08:53.948970    2565 main.go:120] Received NotifyRegistrationStatus call: &RegistrationStatus{PluginRegistered:true,Error:,}

and 

 k logs  ceph-csi-rbd-nodeplugin-fzgjc
I0621 07:05:11.262468       1 controller.go:1337] provision "nginx-example/nginx-logs" class "csi-rbd-sc": started
I0621 07:05:11.262711       1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"nginx-example", Name:"nginx-logs", UID:"a21b17eb-ea57-45b6-8424-faf5c6145e06", APIVersion:"v1", ResourceVersion:"154761538", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "nginx-example/nginx-logs"
I0621 07:05:11.444297       1 controller.go:1442] provision "nginx-example/nginx-logs" class "csi-rbd-sc": volume "pvc-a21b17eb-ea57-45b6-8424-faf5c6145e06" provisioned
I0621 07:05:11.444321       1 controller.go:1455] provision "nginx-example/nginx-logs" class "csi-rbd-sc": succeeded
I0621 07:05:11.482350       1 event.go:285] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"nginx-example", Name:"nginx-logs", UID:"a21b17eb-ea57-45b6-8424-faf5c6145e06", APIVersion:"v1", ResourceVersion:"154761538", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-a21b17eb-ea57-45b6-8424-faf5c6145e06

and external-snapshot controller( k logs -n kube-system snapshot-controller-6cd7b4d677-w27nq ):

I0621 07:30:58.168999       1 leaderelection.go:278] successfully renewed lease kube-system/snapshot-controller-leader
I0621 07:31:03.183456       1 leaderelection.go:278] successfully renewed lease kube-system/snapshot-controller-leader
I0621 07:31:08.200175       1 leaderelection.go:278] successfully renewed lease kube-system/snapshot-controller-leader
I0621 07:31:13.216053       1 leaderelection.go:278] successfully renewed lease kube-system/snapshot-controller-leader
I0621 07:31:15.393517       1 snapshot_controller_base.go:200] enqueued "snapcontent-bdb78dc1-7b6f-4809-9dd3-773d9772ba52" for sync
I0621 07:31:15.393559       1 snapshot_controller_base.go:302] syncContentByKey[snapcontent-bdb78dc1-7b6f-4809-9dd3-773d9772ba52]
I0621 07:31:15.393574       1 util.go:218] storeObjectUpdate updating content "snapcontent-bdb78dc1-7b6f-4809-9dd3-773d9772ba52" with version 154775071
I0621 07:31:15.393582       1 snapshot_controller.go:88] synchronizing VolumeSnapshotContent[snapcontent-bdb78dc1-7b6f-4809-9dd3-773d9772ba52]: content is und to snapshot nginx-example/rbd-pvc-snapshot
I0621 07:31:15.393585       1 snapshot_controller.go:90] syncContent[snapcontent-bdb78dc1-7b6f-4809-9dd3-773d9772ba52]: check if we should add invalid labeon content
I0621 07:31:15.393591       1 snapshot_controller.go:1550] getSnapshotFromStore: snapshot nginx-example/rbd-pvc-snapshot found
I0621 07:31:15.393596       1 snapshot_controller.go:1081] needsUpdateSnapshotStatus[nginx-example/rbd-pvc-snapshot]
I0621 07:31:15.393599       1 snapshot_controller.go:143] synchronizing VolumeSnapshotContent for snapshot [nginx-example/rbd-pvc-snapshot]: update snapshostatus to true if needed.
I0621 07:31:15.393627       1 snapshot_controller_base.go:227] syncSnapshotByKey[nginx-example/rbd-pvc-snapshot]
I0621 07:31:15.393631       1 snapshot_controller_base.go:230] snapshotWorker: snapshot namespace [nginx-example] name [rbd-pvc-snapshot]
I0621 07:31:15.393637       1 snapshot_controller_base.go:353] checkAndUpdateSnapshotClass [rbd-pvc-snapshot]: VolumeSnapshotClassName [csi-rbdplugin-snapcss]
I0621 07:31:15.393640       1 snapshot_controller.go:1284] getSnapshotClass: VolumeSnapshotClassName [csi-rbdplugin-snapclass]
I0621 07:31:15.393646       1 snapshot_controller_base.go:373] VolumeSnapshotClass [csi-rbdplugin-snapclass] Driver [rbd.csi.ceph.com]
I0621 07:31:15.393651       1 snapshot_controller_base.go:247] Updating snapshot "nginx-example/rbd-pvc-snapshot"
I0621 07:31:15.393656       1 snapshot_controller_base.go:383] updateSnapshot "nginx-example/rbd-pvc-snapshot"
I0621 07:31:15.393662       1 util.go:218] storeObjectUpdate updating snapshot "nginx-example/rbd-pvc-snapshot" with version 154772633
I0621 07:31:15.393670       1 snapshot_controller.go:183] synchronizing VolumeSnapshot[nginx-example/rbd-pvc-snapshot]: bound to: "snapcontent-bdb78dc1-7b64809-9dd3-773d9772ba52", Completed: false
I0621 07:31:15.393674       1 snapshot_controller.go:185] syncSnapshot [nginx-example/rbd-pvc-snapshot]: check if we should remove finalizer on snapshot PVsource and remove it if we can
I0621 07:31:15.393682       1 snapshot_controller.go:982] checkandRemovePVCFinalizer for snapshot [rbd-pvc-snapshot]: snapshot status [&v1.VolumeSnapshotStus{BoundVolumeSnapshotContentName:(*string)(0xc00048f040), CreationTime:<nil>, ReadyToUse:(*bool)(0xc0000ba8ea), RestoreSize:(*resource.Quantity)(nil), Err:(*v1.VolumeSnapshotError)(0xc00048f050)}]
I0621 07:31:15.393719       1 snapshot_controller.go:937] Checking isPVCBeingUsed for snapshot [nginx-example/rbd-pvc-snapshot]
I0621 07:31:15.393729       1 snapshot_controller.go:957] Keeping PVC nginx-example/nginx-logs, it is used by snapshot nginx-example/rbd-pvc-snapshot
I0621 07:31:15.393733       1 snapshot_controller.go:194] syncSnapshot[nginx-example/rbd-pvc-snapshot]: check if we should add invalid label on snapshot
I0621 07:31:15.393738       1 snapshot_controller.go:210] syncSnapshot[nginx-example/rbd-pvc-snapshot]: validate snapshot to make sure source has been corrtly specified
I0621 07:31:15.393742       1 snapshot_controller.go:219] syncSnapshot[nginx-example/rbd-pvc-snapshot]: check if we should add finalizers on snapshot
I0621 07:31:15.393748       1 snapshot_controller.go:418] syncUnreadySnapshot nginx-example/rbd-pvc-snapshot
I0621 07:31:15.393751       1 snapshot_controller.go:1298] getSnapshotDriverName: VolumeSnapshot[rbd-pvc-snapshot]
I0621 07:31:15.393753       1 snapshot_controller.go:1284] getSnapshotClass: VolumeSnapshotClassName [csi-rbdplugin-snapclass]
I0621 07:31:15.393763       1 snapshot_controller.go:479] getDynamicallyProvisionedContentFromStore for snapshot nginx-example/rbd-pvc-snapshot
I0621 07:31:15.393776       1 snapshot_controller.go:487] Found VolumeSnapshotContent object snapcontent-bdb78dc1-7b6f-4809-9dd3-773d9772ba52 for snapshot inx-example/rbd-pvc-snapshot
I0621 07:31:15.393780       1 snapshot_controller.go:1047] bindandUpdateVolumeSnapshot for snapshot [rbd-pvc-snapshot]: snapshotContent [snapcontent-bdb78d-7b6f-4809-9dd3-773d9772ba52]
I0621 07:31:15.401809       1 snapshot_controller.go:1057] bindandUpdateVolumeSnapshot [nginx-example/rbd-pvc-snapshot]: trying to update snapshot status
I0621 07:31:15.401827       1 snapshot_controller.go:1113] updateSnapshotStatus[nginx-example/rbd-pvc-snapshot]
I0621 07:31:15.401832       1 snapshot_controller.go:1134] updateSnapshotStatus: updating VolumeSnapshot [&{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:r-pvc-snapshot GenerateName: Namespace:nginx-example SelfLink: UID:bdb78dc1-7b6f-4809-9dd3-773d9772ba52 ResourceVersion:154772633 Generation:1 CreationTimesmp:2023-06-21 07:26:24 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[kubectl.kubernetes.io/last-applied-cfiguration:{"apiVersion":"snapshot.storage.k8s.io/v1","kind":"VolumeSnapshot","metadata":{"annotations":{},"name":"rbd-pvc-snapshot","namespace":"nginx-exale"},"spec":{"source":{"persistentVolumeClaimName":"nginx-logs"},"volumeSnapshotClassName":"csi-rbdplugin-snapclass"}}
] OwnerReferences:[] Finalizers:[snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection snapshot.storage.kubernetes.io/volumesnapshot-bound-proction] ClusterName: ManagedFields:[{Manager:kubectl-client-side-apply Operation:Update APIVersion:snapshot.storage.k8s.io/v1 Time:2023-06-21 07:26:24 +0000TC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:annotations":{".":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}}},"f:spec":{".":{},"f:source{".":{},"f:persistentVolumeClaimName":{}},"f:volumeSnapshotClassName":{}}} Subresource:} {Manager:snapshot-controller Operation:Update APIVersion:snapshot.orage.k8s.io/v1 Time:2023-06-21 07:26:24 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:finalizers":{".":{},"v:\"snapshot.storage.kubernetes.io/vumesnapshot-as-source-protection\"":{},"v:\"snapshot.storage.kubernetes.io/volumesnapshot-bound-protection\"":{}}}} Subresource:} {Manager:snapshot-controlr Operation:Update APIVersion:snapshot.storage.k8s.io/v1 Time:2023-06-21 07:26:25 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:status":{".":{},"f:boundVolumeapshotContentName":{},"f:error":{".":{},"f:message":{},"f:time":{}},"f:readyToUse":{}}} Subresource:status}]} Spec:{Source:{PersistentVolumeClaimName:0xc00a3180 VolumeSnapshotContentName:<nil>} VolumeSnapshotClassName:0xc0004a3190} Status:0xc00059bc80}] based on VolumeSnapshotContentStatus [&{SnapshotHandle:<l> CreationTime:<nil> RestoreSize:<nil> ReadyToUse:0xc0002aa8ef Error:0xc0004a2910}]
I0621 07:31:15.411696       1 util.go:218] storeObjectUpdate updating snapshot "nginx-example/rbd-pvc-snapshot" with version 154772633
I0621 07:31:15.411724       1 snapshot_controller.go:1074] bindandUpdateVolumeSnapshot for snapshot completed [&v1.VolumeSnapshot{TypeMeta:v1.TypeMeta{Kind", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"rbd-pvc-snapshot", GenerateName:"", Namespace:"nginx-example", SelfLink:"", UID:"bdb78dc1-7b6f-4809-9dd3-3d9772ba52", ResourceVersion:"154772633", Generation:1, CreationTimestamp:time.Date(2023, time.June, 21, 7, 26, 24, 0, time.Local), DeletionTimestamp:<nil>DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string{"kubectl.kubernetes.io/last-applied-configuration":""apiVersion\":\"snapshot.storage.k8s.io/v1\",\"kind\":\"VolumeSnapshot\",\"metadata\":{\"annotations\":{},\"name\":\"rbd-pvc-snapshot\",\"namespace\":\"ngi-example\"},\"spec\":{\"source\":{\"persistentVolumeClaimName\":\"nginx-logs\"},\"volumeSnapshotClassName\":\"csi-rbdplugin-snapclass\"}}\n"}, OwnerReferens:[]v1.OwnerReference(nil), Finalizers:[]string{"snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection", "snapshot.storage.kubernetes.io/volumnapshot-bound-protection"}, ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"kubectl-client-side-apply", Operation:"Upde", APIVersion:"snapshot.storage.k8s.io/v1", Time:time.Date(2023, time.June, 21, 7, 26, 24, 0, time.Local), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(c000807518), Subresource:""}, v1.ManagedFieldsEntry{Manager:"snapshot-controller", Operation:"Update", APIVersion:"snapshot.storage.k8s.io/v1", Time:time.De(2023, time.June, 21, 7, 26, 24, 0, time.Local), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc000807548), Subresource:""}, v1.ManagedFieldsEntry{Manar:"snapshot-controller", Operation:"Update", APIVersion:"snapshot.storage.k8s.io/v1", Time:time.Date(2023, time.June, 21, 7, 26, 25, 0, time.Local), Fieldspe:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc000807578), Subresource:"status"}}}, Spec:v1.VolumeSnapshotSpec{Source:v1.VolumeSnapshotSource{PersistentVolumeClmName:(*string)(0xc00059d900), VolumeSnapshotContentName:(*string)(nil)}, VolumeSnapshotClassName:(*string)(0xc00059d910)}, Status:(*v1.VolumeSnapshotStatu(0xc000211110)}]
I0621 07:31:15.411829       1 snapshot_controller.go:497] bindandUpdateVolumeSnapshot &{{ } {rbd-pvc-snapshot  nginx-example  bdb78dc1-7b6f-4809-9dd3-773d92ba52 154772633 1 2023-06-21 07:26:24 +0000 UTC <nil> <nil> map[] map[kubectl.kubernetes.io/last-applied-configuration:{"apiVersion":"snapshot.storage.k8s./v1","kind":"VolumeSnapshot","metadata":{"annotations":{},"name":"rbd-pvc-snapshot","namespace":"nginx-example"},"spec":{"source":{"persistentVolumeClaimNa":"nginx-logs"},"volumeSnapshotClassName":"csi-rbdplugin-snapclass"}}
] [] [snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection snapshot.storage.kubernetes.io/volumesnapshot-bound-protection]  [{kubectl-client-de-apply Update snapshot.storage.k8s.io/v1 2023-06-21 07:26:24 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:kubectl.kubernetes.io/last-appld-configuration":{}}},"f:spec":{".":{},"f:source":{".":{},"f:persistentVolumeClaimName":{}},"f:volumeSnapshotClassName":{}}} } {snapshot-controller Update apshot.storage.k8s.io/v1 2023-06-21 07:26:24 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{".":{},"v:\"snapshot.storage.kubernetes.io/volumesnapshot-asource-protection\"":{},"v:\"snapshot.storage.kubernetes.io/volumesnapshot-bound-protection\"":{}}}} } {snapshot-controller Update snapshot.storage.k8s.io/v2023-06-21 07:26:25 +0000 UTC FieldsV1 {"f:status":{".":{},"f:boundVolumeSnapshotContentName":{},"f:error":{".":{},"f:message":{},"f:time":{}},"f:readyToUs:{}}} status}]} {{0xc00059d900 <nil>} 0xc00059d910} 0xc000211110}
I0621 07:31:15.436513       1 snapshot_controller_base.go:200] enqueued "snapcontent-bdb78dc1-7b6f-4809-9dd3-773d9772ba52" for sync
I0621 07:31:15.436594       1 snapshot_controller_base.go:302] syncContentByKey[snapcontent-bdb78dc1-7b6f-4809-9dd3-773d9772ba52]
I0621 07:31:15.436609       1 util.go:218] storeObjectUpdate updating content "snapcontent-bdb78dc1-7b6f-4809-9dd3-773d9772ba52" with version 154775072
I0621 07:31:15.436620       1 snapshot_controller.go:88] synchronizing VolumeSnapshotContent[snapcontent-bdb78dc1-7b6f-4809-9dd3-773d9772ba52]: content is und to snapshot nginx-example/rbd-pvc-snapshot
I0621 07:31:15.436630       1 snapshot_controller.go:90] syncContent[snapcontent-bdb78dc1-7b6f-4809-9dd3-773d9772ba52]: check if we should add invalid labeon content
I0621 07:31:15.436638       1 snapshot_controller.go:1550] getSnapshotFromStore: snapshot nginx-example/rbd-pvc-snapshot found
I0621 07:31:15.436651       1 snapshot_controller.go:1081] needsUpdateSnapshotStatus[nginx-example/rbd-pvc-snapshot]
I0621 07:31:15.436656       1 snapshot_controller.go:143] synchronizing VolumeSnapshotContent for snapshot [nginx-example/rbd-pvc-snapshot]: update snapshostatus to true if needed.
I0621 07:31:15.436687       1 snapshot_controller_base.go:227] syncSnapshotByKey[nginx-example/rbd-pvc-snapshot]
I0621 07:31:15.436696       1 snapshot_controller_base.go:230] snapshotWorker: snapshot namespace [nginx-example] name [rbd-pvc-snapshot]
I0621 07:31:15.436705       1 snapshot_controller_base.go:353] checkAndUpdateSnapshotClass [rbd-pvc-snapshot]: VolumeSnapshotClassName [csi-rbdplugin-snapcss]
I0621 07:31:15.436711       1 snapshot_controller.go:1284] getSnapshotClass: VolumeSnapshotClassName [csi-rbdplugin-snapclass]
I0621 07:31:15.436724       1 snapshot_controller_base.go:373] VolumeSnapshotClass [csi-rbdplugin-snapclass] Driver [rbd.csi.ceph.com]
I0621 07:31:15.436733       1 snapshot_controller_base.go:247] Updating snapshot "nginx-example/rbd-pvc-snapshot"
I0621 07:31:15.436745       1 snapshot_controller_base.go:383] updateSnapshot "nginx-example/rbd-pvc-snapshot"
I0621 07:31:15.436754       1 util.go:218] storeObjectUpdate updating snapshot "nginx-example/rbd-pvc-snapshot" with version 154772633
I0621 07:31:15.436770       1 snapshot_controller.go:183] synchronizing VolumeSnapshot[nginx-example/rbd-pvc-snapshot]: bound to: "snapcontent-bdb78dc1-7b64809-9dd3-773d9772ba52", Completed: false
I0621 07:31:15.436782       1 snapshot_controller.go:185] syncSnapshot [nginx-example/rbd-pvc-snapshot]: check if we should remove finalizer on snapshot PVsource and remove it if we can
I0621 07:31:15.436793       1 snapshot_controller.go:982] checkandRemovePVCFinalizer for snapshot [rbd-pvc-snapshot]: snapshot status [&v1.VolumeSnapshotStus{BoundVolumeSnapshotContentName:(*string)(0xc00048f040), CreationTime:<nil>, ReadyToUse:(*bool)(0xc0000ba8ea), RestoreSize:(*resource.Quantity)(nil), Err:(*v1.VolumeSnapshotError)(0xc00048f050)}]
I0621 07:31:15.436861       1 snapshot_controller.go:937] Checking isPVCBeingUsed for snapshot [nginx-example/rbd-pvc-snapshot]
I0621 07:31:15.436876       1 snapshot_controller.go:957] Keeping PVC nginx-example/nginx-logs, it is used by snapshot nginx-example/rbd-pvc-snapshot
I0621 07:31:15.436887       1 snapshot_controller.go:194] syncSnapshot[nginx-example/rbd-pvc-snapshot]: check if we should add invalid label on snapshot
I0621 07:31:15.436894       1 snapshot_controller.go:210] syncSnapshot[nginx-example/rbd-pvc-snapshot]: validate snapshot to make sure source has been corrtly specified
I0621 07:31:15.436905       1 snapshot_controller.go:219] syncSnapshot[nginx-example/rbd-pvc-snapshot]: check if we should add finalizers on snapshot
I0621 07:31:15.436913       1 snapshot_controller.go:418] syncUnreadySnapshot nginx-example/rbd-pvc-snapshot
I0621 07:31:15.436922       1 snapshot_controller.go:1298] getSnapshotDriverName: VolumeSnapshot[rbd-pvc-snapshot]
I0621 07:31:15.436927       1 snapshot_controller.go:1284] getSnapshotClass: VolumeSnapshotClassName [csi-rbdplugin-snapclass]
I0621 07:31:15.436944       1 snapshot_controller.go:479] getDynamicallyProvisionedContentFromStore for snapshot nginx-example/rbd-pvc-snapshot
I0621 07:31:15.436959       1 snapshot_controller.go:487] Found VolumeSnapshotContent object snapcontent-bdb78dc1-7b6f-4809-9dd3-773d9772ba52 for snapshot inx-example/rbd-pvc-snapshot
I0621 07:31:15.436967       1 snapshot_controller.go:1047] bindandUpdateVolumeSnapshot for snapshot [rbd-pvc-snapshot]: snapshotContent [snapcontent-bdb78d-7b6f-4809-9dd3-773d9772ba52]
I0621 07:31:15.441385       1 snapshot_controller_base.go:200] enqueued "snapcontent-bdb78dc1-7b6f-4809-9dd3-773d9772ba52" for sync
I0621 07:31:15.441410       1 snapshot_controller_base.go:302] syncContentByKey[snapcontent-bdb78dc1-7b6f-4809-9dd3-773d9772ba52]
I0621 07:31:15.441421       1 util.go:218] storeObjectUpdate updating content "snapcontent-bdb78dc1-7b6f-4809-9dd3-773d9772ba52" with version 154775074
I0621 07:31:15.441429       1 snapshot_controller.go:88] synchronizing VolumeSnapshotContent[snapcontent-bdb78dc1-7b6f-4809-9dd3-773d9772ba52]: content is und to snapshot nginx-example/rbd-pvc-snapshot
I0621 07:31:15.441433       1 snapshot_controller.go:90] syncContent[snapcontent-bdb78dc1-7b6f-4809-9dd3-773d9772ba52]: check if we should add invalid labeon content
I0621 07:31:15.441524       1 snapshot_controller.go:1550] getSnapshotFromStore: snapshot nginx-example/rbd-pvc-snapshot found
I0621 07:31:15.441535       1 snapshot_controller.go:1081] needsUpdateSnapshotStatus[nginx-example/rbd-pvc-snapshot]
I0621 07:31:15.441540       1 snapshot_controller.go:143] synchronizing VolumeSnapshotContent for snapshot [nginx-example/rbd-pvc-snapshot]: update snapshostatus to true if needed.
I0621 07:31:15.442529       1 snapshot_controller.go:1057] bindandUpdateVolumeSnapshot [nginx-example/rbd-pvc-snapshot]: trying to update snapshot status
I0621 07:31:15.442549       1 snapshot_controller.go:1113] updateSnapshotStatus[nginx-example/rbd-pvc-snapshot]
I0621 07:31:15.442555       1 snapshot_controller.go:1134] updateSnapshotStatus: updating VolumeSnapshot [&{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:r-pvc-snapshot GenerateName: Namespace:nginx-example SelfLink: UID:bdb78dc1-7b6f-4809-9dd3-773d9772ba52 ResourceVersion:154772633 Generation:1 CreationTimesmp:2023-06-21 07:26:24 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[kubectl.kubernetes.io/last-applied-cfiguration:{"apiVersion":"snapshot.storage.k8s.io/v1","kind":"VolumeSnapshot","metadata":{"annotations":{},"name":"rbd-pvc-snapshot","namespace":"nginx-exale"},"spec":{"source":{"persistentVolumeClaimName":"nginx-logs"},"volumeSnapshotClassName":"csi-rbdplugin-snapclass"}}
] OwnerReferences:[] Finalizers:[snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection snapshot.storage.kubernetes.io/volumesnapshot-bound-proction] ClusterName: ManagedFields:[{Manager:kubectl-client-side-apply Operation:Update APIVersion:snapshot.storage.k8s.io/v1 Time:2023-06-21 07:26:24 +0000TC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:annotations":{".":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}}},"f:spec":{".":{},"f:source{".":{},"f:persistentVolumeClaimName":{}},"f:volumeSnapshotClassName":{}}} Subresource:} {Manager:snapshot-controller Operation:Update APIVersion:snapshot.orage.k8s.io/v1 Time:2023-06-21 07:26:24 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:finalizers":{".":{},"v:\"snapshot.storage.kubernetes.io/vumesnapshot-as-source-protection\"":{},"v:\"snapshot.storage.kubernetes.io/volumesnapshot-bound-protection\"":{}}}} Subresource:} {Manager:snapshot-controlr Operation:Update APIVersion:snapshot.storage.k8s.io/v1 Time:2023-06-21 07:26:25 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:status":{".":{},"f:boundVolumeapshotContentName":{},"f:error":{".":{},"f:message":{},"f:time":{}},"f:readyToUse":{}}} Subresource:status}]} Spec:{Source:{PersistentVolumeClaimName:0xc00a6800 VolumeSnapshotContentName:<nil>} VolumeSnapshotClassName:0xc0002a6810} Status:0xc0004a0fc0}] based on VolumeSnapshotContentStatus [&{SnapshotHandle:<l> CreationTime:<nil> RestoreSize:<nil> ReadyToUse:0xc0004a7ed9 Error:0xc00059de20}]
I0621 07:31:15.460057       1 util.go:218] storeObjectUpdate updating snapshot "nginx-example/rbd-pvc-snapshot" with version 154772633
I0621 07:31:15.460089       1 snapshot_controller.go:1074] bindandUpdateVolumeSnapshot for snapshot completed [&v1.VolumeSnapshot{TypeMeta:v1.TypeMeta{Kind", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"rbd-pvc-snapshot", GenerateName:"", Namespace:"nginx-example", SelfLink:"", UID:"bdb78dc1-7b6f-4809-9dd3-3d9772ba52", ResourceVersion:"154772633", Generation:1, CreationTimestamp:time.Date(2023, time.June, 21, 7, 26, 24, 0, time.Local), DeletionTimestamp:<nil>DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string{"kubectl.kubernetes.io/last-applied-configuration":""apiVersion\":\"snapshot.storage.k8s.io/v1\",\"kind\":\"VolumeSnapshot\",\"metadata\":{\"annotations\":{},\"name\":\"rbd-pvc-snapshot\",\"namespace\":\"ngi-example\"},\"spec\":{\"source\":{\"persistentVolumeClaimName\":\"nginx-logs\"},\"volumeSnapshotClassName\":\"csi-rbdplugin-snapclass\"}}\n"}, OwnerReferens:[]v1.OwnerReference(nil), Finalizers:[]string{"snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection", "snapshot.storage.kubernetes.io/volumnapshot-bound-protection"}, ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"kubectl-client-side-apply", Operation:"Upde", APIVersion:"snapshot.storage.k8s.io/v1", Time:time.Date(2023, time.June, 21, 7, 26, 24, 0, time.Local), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(c000567188), Subresource:""}, v1.ManagedFieldsEntry{Manager:"snapshot-controller", Operation:"Update", APIVersion:"snapshot.storage.k8s.io/v1", Time:time.De(2023, time.June, 21, 7, 26, 24, 0, time.Local), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc0005671b8), Subresource:""}, v1.ManagedFieldsEntry{Manar:"snapshot-controller", Operation:"Update", APIVersion:"snapshot.storage.k8s.io/v1", Time:time.Date(2023, time.June, 21, 7, 26, 25, 0, time.Local), Fieldspe:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc0005671e8), Subresource:"status"}}}, Spec:v1.VolumeSnapshotSpec{Source:v1.VolumeSnapshotSource{PersistentVolumeClmName:(*string)(0xc0004a39c0), VolumeSnapshotContentName:(*string)(nil)}, VolumeSnapshotClassName:(*string)(0xc0004a39d0)}, Status:(*v1.VolumeSnapshotStatu(0xc00003a900)}]
I0621 07:31:15.460197       1 snapshot_controller.go:497] bindandUpdateVolumeSnapshot &{{ } {rbd-pvc-snapshot  nginx-example  bdb78dc1-7b6f-4809-9dd3-773d92ba52 154772633 1 2023-06-21 07:26:24 +0000 UTC <nil> <nil> map[] map[kubectl.kubernetes.io/last-applied-configuration:{"apiVersion":"snapshot.storage.k8s./v1","kind":"VolumeSnapshot","metadata":{"annotations":{},"name":"rbd-pvc-snapshot","namespace":"nginx-example"},"spec":{"source":{"persistentVolumeClaimNa":"nginx-logs"},"volumeSnapshotClassName":"csi-rbdplugin-snapclass"}}
] [] [snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection snapshot.storage.kubernetes.io/volumesnapshot-bound-protection]  [{kubectl-client-de-apply Update snapshot.storage.k8s.io/v1 2023-06-21 07:26:24 +0000 UTC FieldsV1 {"f:metadata":{"f:annotations":{".":{},"f:kubectl.kubernetes.io/last-appld-configuration":{}}},"f:spec":{".":{},"f:source":{".":{},"f:persistentVolumeClaimName":{}},"f:volumeSnapshotClassName":{}}} } {snapshot-controller Update apshot.storage.k8s.io/v1 2023-06-21 07:26:24 +0000 UTC FieldsV1 {"f:metadata":{"f:finalizers":{".":{},"v:\"snapshot.storage.kubernetes.io/volumesnapshot-asource-protection\"":{},"v:\"snapshot.storage.kubernetes.io/volumesnapshot-bound-protection\"":{}}}} } {snapshot-controller Update snapshot.storage.k8s.io/v2023-06-21 07:26:25 +0000 UTC FieldsV1 {"f:status":{".":{},"f:boundVolumeSnapshotContentName":{},"f:error":{".":{},"f:message":{},"f:time":{}},"f:readyToUs:{}}} status}]} {{0xc0004a39c0 <nil>} 0xc0004a39d0} 0xc00003a900}
I0621 07:31:15.460286       1 snapshot_controller_base.go:227] syncSnapshotByKey[nginx-example/rbd-pvc-snapshot]
I0621 07:31:15.460294       1 snapshot_controller_base.go:230] snapshotWorker: snapshot namespace [nginx-example] name [rbd-pvc-snapshot]
I0621 07:31:15.460303       1 snapshot_controller_base.go:353] checkAndUpdateSnapshotClass [rbd-pvc-snapshot]: VolumeSnapshotClassName [csi-rbdplugin-snapcss]
I0621 07:31:15.460316       1 snapshot_controller.go:1284] getSnapshotClass: VolumeSnapshotClassName [csi-rbdplugin-snapclass]
I0621 07:31:15.460326       1 snapshot_controller_base.go:373] VolumeSnapshotClass [csi-rbdplugin-snapclass] Driver [rbd.csi.ceph.com]
I0621 07:31:15.460338       1 snapshot_controller_base.go:247] Updating snapshot "nginx-example/rbd-pvc-snapshot"
I0621 07:31:15.460350       1 snapshot_controller_base.go:383] updateSnapshot "nginx-example/rbd-pvc-snapshot"
I0621 07:31:15.460361       1 util.go:218] storeObjectUpdate updating snapshot "nginx-example/rbd-pvc-snapshot" with version 154772633
I0621 07:31:15.460376       1 snapshot_controller.go:183] synchronizing VolumeSnapshot[nginx-example/rbd-pvc-snapshot]: bound to: "snapcontent-bdb78dc1-7b64809-9dd3-773d9772ba52", Completed: false

and the result:

 k get volumesnapshot
NAME               READYTOUSE   SOURCEPVC    SOURCESNAPSHOTCONTENT   RESTORESIZE   SNAPSHOTCLASS             SNAPSHOTCONTENT                                  CREATIONTIME   AGE
rbd-pvc-snapshot   false        nginx-logs                                         csi-rbdplugin-snapclass   snapcontent-bdb78dc1-7b6f-4809-9dd3-773d9772ba                  91s
.
.
.
k get volumesnapshot
NAME               READYTOUSE   SOURCEPVC    SOURCESNAPSHOTCONTENT   RESTORESIZE   SNAPSHOTCLASS             SNAPSHOTCONTENT                                    CREATIONTIME   AGE
rbd-pvc-snapshot   false        nginx-logs                                         csi-rbdplugin-snapclass   snapcontent-bdb78dc1-7b6f-4809-9dd3-773d9772ba

and lets see describe:

 k describe  volumesnapshot rbd-pvc-snapshot
Name:         rbd-pvc-snapshot
Namespace:    nginx-example
Labels:       <none>
Annotations:  <none>
API Version:  snapshot.storage.k8s.io/v1
Kind:         VolumeSnapshot
Metadata:
  Creation Timestamp:  2023-06-21T07:26:24Z
  Finalizers:
    snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection
    snapshot.storage.kubernetes.io/volumesnapshot-bound-protection
  Generation:  1
  Managed Fields:
    API Version:  snapshot.storage.k8s.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        .:
        f:source:
          .:
          f:persistentVolumeClaimName:
        f:volumeSnapshotClassName:
    Manager:      kubectl-client-side-apply
    Operation:    Update
    Time:         2023-06-21T07:26:24Z
    API Version:  snapshot.storage.k8s.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
          .:
          v:"snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection":
          v:"snapshot.storage.kubernetes.io/volumesnapshot-bound-protection":
    Manager:      snapshot-controller
    Operation:    Update
    Time:         2023-06-21T07:26:24Z
    API Version:  snapshot.storage.k8s.io/v1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:boundVolumeSnapshotContentName:
        f:error:
          .:
          f:message:
          f:time:
        f:readyToUse:
    Manager:         snapshot-controller
    Operation:       Update
    Subresource:     status
    Time:            2023-06-21T07:26:25Z
  Resource Version:  154772633
  UID:               bdb78dc1-7b6f-4809-9dd3-773d9772ba52
Spec:
  Source:
    Persistent Volume Claim Name:  nginx-logs
  Volume Snapshot Class Name:      csi-rbdplugin-snapclass
Status:
  Bound Volume Snapshot Content Name:  snapcontent-bdb78dc1-7b6f-4809-9dd3-773d9772ba52
  Error:
    Message:     Failed to check and update snapshot content: failed to take snapshot of the volume 0001-0024-275ae542-d1f9-11ed-893f-005056ad62c4-0000000000000002-b2558284-70b0-4f86-9a03-c32f9ef60b71: "rpc error: code = Internal desc = failed to fetch monitor list using clusterID (b7bd17b4-98a2-46e2-8c14-c0a8d54f33ad): missing configuration for cluster ID \"b7bd17b4-98a2-46e2-8c14-c0a8d54f33ad\""
    Time:        2023-06-21T07:26:25Z
  Ready To Use:  false
Events:
  Type    Reason            Age   From                 Message
  ----    ------            ----  ----                 -------
  Normal  CreatingSnapshot  19m   snapshot-controller  Waiting for a snapshot nginx-example/rbd-pvc-snapshot to be created by the CSI driver.
Borosan commented 1 year ago

i also noticed, i have another problem! 

k get sc csi-rbd-sc -oyaml
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    meta.helm.sh/release-name: ceph-csi-rbd
    meta.helm.sh/release-namespace: ceph-csi-rbd
  creationTimestamp: "2023-04-04T11:48:06Z"
  labels:
    app: ceph-csi-rbd
    app.kubernetes.io/managed-by: Helm
    chart: ceph-csi-rbd-3.8.0
    heritage: Helm
    release: ceph-csi-rbd
  name: csi-rbd-sc
  resourceVersion: "98104911"
  uid: b7bd17b4-98a2-46e2-8c14-c0a8d54f33ad
mountOptions:
- discard
parameters:
  clusterID: 275ae542-d1f9-11ed-893f-005056ad62c4
  csi.storage.k8s.io/controller-expand-secret-name: csi-rbd-secret
  csi.storage.k8s.io/controller-expand-secret-namespace: ceph-csi-rbd
  csi.storage.k8s.io/fstype: ext4
  csi.storage.k8s.io/node-stage-secret-name: csi-rbd-secret
  csi.storage.k8s.io/node-stage-secret-namespace: ceph-csi-rbd
  csi.storage.k8s.io/provisioner-secret-name: csi-rbd-secret
  csi.storage.k8s.io/provisioner-secret-namespace: ceph-csi-rbd
  imageFeatures: layering
  pool: stgk1poolrep1
provisioner: rbd.csi.ceph.com
reclaimPolicy: Delete
volumeBindingMode: Immediate

and about my volume snapshot class:

k get volumesnapshot rbd-pvc-snapshot  -oyaml
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"snapshot.storage.k8s.io/v1","kind":"VolumeSnapshot","metadata":{"annotations":{},"name":"rbd-pvc-snapshot","namespace":"nginx-example"},"spec":{"source":{"persistentVolumeClaimName":"nginx-logs"},"volumeSnapshotClassName":"csi-rbdplugin-snapclass"}}
  creationTimestamp: "2023-06-21T07:26:24Z"
  finalizers:
  - snapshot.storage.kubernetes.io/volumesnapshot-as-source-protection
  - snapshot.storage.kubernetes.io/volumesnapshot-bound-protection
  generation: 1
  name: rbd-pvc-snapshot
  namespace: nginx-example
  resourceVersion: "154772633"
  uid: bdb78dc1-7b6f-4809-9dd3-773d9772ba52
spec:
  source:
    persistentVolumeClaimName: nginx-logs
  volumeSnapshotClassName: csi-rbdplugin-snapclass
status:
  boundVolumeSnapshotContentName: snapcontent-bdb78dc1-7b6f-4809-9dd3-773d9772ba52
  error:
    message: 'Failed to check and update snapshot content: failed to take snapshot
      of the volume 0001-0024-275ae542-d1f9-11ed-893f-005056ad62c4-0000000000000002-b2558284-70b0-4f86-9a03-c32f9ef60b71:
      "rpc error: code = Internal desc = failed to fetch monitor list using clusterID
      (b7bd17b4-98a2-46e2-8c14-c0a8d54f33ad): missing configuration for cluster ID
      \"b7bd17b4-98a2-46e2-8c14-c0a8d54f33ad\""'
    time: "2023-06-21T07:26:25Z"
  readyToUse: false
Madhu-1 commented 1 year ago

Okay, still, you have not provided the required logs from the provisioner pod.

Message: Failed to check and update snapshot content: failed to take snapshot of the volume 0001-0024-275ae542-d1f9-11ed-893f-005056ad62c4-0000000000000002-b2558284-70b0-4f86-9a03-c32f9ef60b71: "rpc error: code = Internal desc = failed to fetch monitor list using clusterID (b7bd17b4-98a2-46e2-8c14-c0a8d54f33ad): missing configuration for cluster ID \"b7bd17b4-98a2-46e2-8c14-c0a8d54f33ad\""

Do you have any clusterID with this ID in the configmap? can you check this?

Borosan commented 1 year ago

thank you and right now i was working on that. seems the problem is resolved. i forget to check volume snap shot as i send it to you above. be cause i had problems with longhorn and had that background! :-) sorry for that and thank you. the problem resolved.