VolumeSnapshot never gets READYTOUSE=true

[jsalatiel]

What steps did you take and what happened: [A clear and concise description of what the bug is, and what commands you ran.] Can't create a valid snapshot from a openebs-lvm pvc. The volumesnapshot never gets READYTOUSE=true

What did you expect to happen: Snapshot should be created and available for use

The output of the following commands will help us better understand what's going on: (Pasting long output into a GitHub gist or other Pastebin is fine.)

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.] Installed like this: ( note the --set lvm-localpv.crd.volumeSnapshot=False because I already have those crds installed for longhorn)

helm install openebs openebs/openebs --set cstor.enabled=false --set nfs-provisioner.enabled=false --set localprovisioner.enabled=false --set zfs-localpv.enabled=false --set lvm-localpv.enabled=true  --set ndmOperator.enabled=false  --set ndm.enabled=false --set lvm-localpv.crd.volumeSnapshot=False  --namespace kube-system

storageclass like this:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: openebs-lvm
  annotations:
    storageclass.kubernetes.io/is-default-class: "false"
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
parameters:
  storage: "lvm"
  volgroup: "system"
  fsType: "xfs"
provisioner: local.csi.openebs.io
allowedTopologies:
- matchLabelExpressions:
  - key: kubernetes.io/hostname
    values:
      - node0

Snapshot class like this:

kind: VolumeSnapshotClass
apiVersion: snapshot.storage.k8s.io/v1
metadata:
  name: lvmpv-snapclass
  annotations:
    snapshot.storage.kubernetes.io/is-default-class: "true"
driver: local.csi.openebs.io
deletionPolicy: Delete

VolumeSnapshot like this:

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: lvm-localpv-snap
spec:
  volumeSnapshotClassName: lvmpv-snapclass
  source:
    persistentVolumeClaimName: wppvc

The logs only shows this:

5m42s       Warning   SnapshotFinalizerError          volumesnapshot/lvm-localpv-snap              Failed to check and update snapshot: snapshot controller failed to update kube-system/lvm-localpv-snap on API server: Operation cannot be fulfilled on volumesnapshots.snapshot.storage.k8s.io "lvm-localpv-snap": the object has been modified; please apply your changes to the latest version and try again
5m42s       Warning   SnapshotContentCreationFailed   volumesnapshot/lvm-localpv-snap              Failed to create snapshot content with error snapshot controller failed to update wppvc on API server: Operation cannot be fulfilled on persistentvolumeclaims "wppvc": the object has been modified; please apply your changes to the latest version and try again
5m42s       Normal    CreatingSnapshot                volumesnapshot/lvm-localpv-snap              Waiting for a snapshot kube-system/lvm-localpv-snap to be created by the CSI driver.
5m42s       Normal    CreatingSnapshot                volumesnapshot/lvm-localpv-snap              Waiting for a snapshot kube-system/lvm-localpv-snap to be created by the CSI driver.
5m42s       Normal    SnapshotCreated                 volumesnapshot/lvm-localpv-snap              Snapshot kube-system/lvm-localpv-snap was successfully created by the CSI driver.

The VolumeSnapshot and VolumeSnapshotContent are forever like this:

NAME                                                      READYTOUSE   SOURCEPVC   SOURCESNAPSHOTCONTENT   RESTORESIZE   SNAPSHOTCLASS     SNAPSHOTCONTENT                                    CREATIONTIME   AGE
volumesnapshot.snapshot.storage.k8s.io/lvm-localpv-snap   false        wppvc                               0             lvmpv-snapclass   snapcontent-5edc103b-0748-4eb6-899f-4e7305c571a6   6m41s          6m41s

NAME                                                                                             READYTOUSE   RESTORESIZE   DELETIONPOLICY   DRIVER                 VOLUMESNAPSHOTCLASS   VOLUMESNAPSHOT     VOLUMESNAPSHOTNAMESPACE   AGE
volumesnapshotcontent.snapshot.storage.k8s.io/snapcontent-5edc103b-0748-4eb6-899f-4e7305c571a6   false        0             Delete           local.csi.openebs.io   lvmpv-snapclass       lvm-localpv-snap   kube-system               6m41s

Environment:

• LVM Driver version
• Kubernetes version (use kubectl version): v1.26.5
• Kubernetes installer & version: kubespray
• Cloud provider or hardware configuration: baremetal
• OS (e.g. from /etc/os-release): almalinux 9.2

[abhilashshetty04]

Hi @jsalatiel , Thanks for raising the issue. Could you please share us logs of lvm csi controller container from openebs-lvm-controller-0 pod and lvm csi node pod which runs on node0

Is this still reproducible?

[jsalatiel]

I do not have access to that cluster anymore, but I have created a new single node cluster and could easily reproduce:

Here are the logs for the controller and the csi-snapshotter container.

[13567436138]

How to solve

[dsharma-dc]

@jsalatiel Please check if you have dm_snapshot module loaded in your kernel.

lsmod | grep dm_snapshot

If not, please install it by doing modprobe and retry.

[dsharma-dc]

Create snapshot, it's not ReadyToUse

$ kubectl get vs
NAME         READYTOUSE   SOURCEPVC      SOURCESNAPSHOTCONTENT   RESTORESIZE   SNAPSHOTCLASS     SNAPSHOTCONTENT                                    CREATIONTIME   AGE
dslvm-snap   false        csi-lvmpv-ds                           0             lvmpv-snapclass   snapcontent-46f801a2-2bc4-4748-9936-670994cf67f4   10s            10s

Install dm_snapshot module

$ sudo modprobe dm_snapshot

Check volume snapshot again

$ kubectl get vs
NAME         READYTOUSE   SOURCEPVC      SOURCESNAPSHOTCONTENT   RESTORESIZE   SNAPSHOTCLASS     SNAPSHOTCONTENT                                    CREATIONTIME   AGE
dslvm-snap   true         csi-lvmpv-ds                           0             lvmpv-snapclass   snapcontent-46f801a2-2bc4-4748-9936-670994cf67f4   40s            40s

[dsharma-dc]

Closing as per above comment. Please feel free to reopen if still facing any issue.

[dsharma-dc]

@Abhinandan-Purkait @abhilashshetty04 Shall we mention above module requirement as part of documentation?

[dsharma-dc]

Ok I just noticed it's already mentioned in documentation under Prerequisites section.