search

openebs / lvm-localpv

Dynamically provision Stateful Persistent Node-Local Volumes & Filesystems for Kubernetes that is integrated with a backend LVM2 data storage stack.
Apache License 2.0
245 stars 92 forks source link

Snapshots are not created #190

Closed benn0r closed 2 years ago

benn0r commented 2 years ago

What steps did you take and what happened: Followed this guide: https://github.com/openebs/lvm-localpv/blob/develop/docs/snapshot.md

What did you expect to happen: Snapshot should be created

The output of the following commands will help us better understand what's going on:

E0501 15:06:50.777462 1 volume.go:270] Get snapshot failed err: lvmsnapshots.local.openebs.io "snapshot-44c1fe7d-f202-4af5-b56e-69081360e95e" not found E0501 15:06:50.780672 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to handle CreateSnapshotRequest for pvc-01f43051-83af-409f-922a-9a3653351aad: snapshot-44c1fe7d-f202-4af5-b56e-69081360e95e, {LVMSnapshot.local.openebs.io "snapshot-44c1fe7d-f202-4af5-b56e-69081360e95e" is invalid: [spec.capacity: Required value, spec.vgPattern: Required value]}

Anything else you would like to add:

I can create snapshots manually without any problems: lvcreate --size 1G --snapshot --name my-name /dev/tank/pvc-01f43051-83af-409f-922a-9a3653351aad In the log from the controller i see this name for the snapshot: snapshot-44c1fe7d-f202-4af5-b56e-69081360e95e. When trying to manually create this snapshot i get this error: Names starting "snapshot" are reserved. Please choose a different LV name. Could this be the issue? How to fix that?

I have checked all logs but i cannot find any useful information about why creating the snapshot fails

pawanpraka1 commented 2 years ago

Can you share the snapshot yaml and also share the lvm volume yaml you are using to create the snapshot?

benn0r commented 2 years ago

Here my yaml, i hope it helps:

kind: VolumeSnapshotClass
apiVersion: snapshot.storage.k8s.io/v1
metadata:
  name: lvmpv-snapclass
  annotations:
    snapshot.storage.kubernetes.io/is-default-class: "true"
driver: local.csi.openebs.io
deletionPolicy: Delete
parameters:
  snapSize: 50%
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: test-snap
spec:
  volumeSnapshotClassName: lvmpv-snapclass
  source:
    persistentVolumeClaimName: data-test-mysql-simple-0
apiVersion: local.openebs.io/v1alpha1
kind: LVMVolume
metadata:
  creationTimestamp: "2022-05-02T07:42:54Z"
  finalizers:
  - lvm.openebs.io/finalizer
  generation: 3
  labels:
    kubernetes.io/nodename: k8s-st-node
  managedFields:
  - apiVersion: local.openebs.io/v1alpha1
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
          .: {}
          v:"lvm.openebs.io/finalizer": {}
        f:labels:
          .: {}
          f:kubernetes.io/nodename: {}
      f:spec:
        .: {}
        f:capacity: {}
        f:ownerNodeID: {}
        f:shared: {}
        f:thinProvision: {}
        f:vgPattern: {}
        f:volGroup: {}
      f:status:
        .: {}
        f:state: {}
    manager: lvm-driver
    operation: Update
    time: "2022-05-02T07:42:54Z"
  name: pvc-81fb10f6-8b34-408f-a3ab-38faeb9c662a
  namespace: default
  resourceVersion: "5647759"
  uid: 5a574433-e320-45e0-bfb1-c5f015393b23
spec:
  capacity: "5368709120"
  ownerNodeID: k8s-st-node
  shared: "yes"
  thinProvision: "no"
  vgPattern: ^tank$
  volGroup: tank
status:
  state: Ready
pawanpraka1 commented 2 years ago

@benn0r could you share the controller logs and the daemonset logs of the node where volume is present?

benn0r commented 2 years ago

daemonset:

Mon, May 2 2022 9:40:13 am | I0502 07:40:13.294372 1 lvmnode.go:274] Successfully synced 'default/k8s-st-node'
Mon, May 2 2022 9:41:13 am | I0502 07:41:13.286222 1 lvmnode.go:274] Successfully synced 'default/k8s-st-node'
Mon, May 2 2022 9:42:13 am | I0502 07:42:13.286282 1 lvmnode.go:274] Successfully synced 'default/k8s-st-node'
Mon, May 2 2022 9:42:54 am | I0502 07:42:54.743033 1 volume.go:209] Got add event for Vol pvc-81fb10f6-8b34-408f-a3ab-38faeb9c662a
Mon, May 2 2022 9:42:54 am | I0502 07:42:54.870493 1 lvm_util.go:193] lvm: created volume tank/pvc-81fb10f6-8b34-408f-a3ab-38faeb9c662a
Mon, May 2 2022 9:42:54 am | I0502 07:42:54.874348 1 volume.go:337] Successfully synced 'default/pvc-81fb10f6-8b34-408f-a3ab-38faeb9c662a'
Mon, May 2 2022 9:42:56 am | I0502 07:42:56.859746 1 grpc.go:72] GRPC call: /csi.v1.Node/NodePublishVolume requests {"target_path":"/var/lib/kubelet/pods/48d32d60-ca98-4861-83af-e1087eeb4673/volumes/kubernetes.io~csi/pvc-81fb10f6-8b34-408f-a3ab-38faeb9c662a/mount","volume_capability":{"AccessType":{"Mount":{"fs_type":"ext4"}},"access_mode":{"mode":1}},"volume_context":{"csi.storage.k8s.io/ephemeral":"false","csi.storage.k8s.io/pod.name":"test-mysql-simple-0","csi.storage.k8s.io/pod.namespace":"default","csi.storage.k8s.io/pod.uid":"48d32d60-ca98-4861-83af-e1087eeb4673","csi.storage.k8s.io/serviceAccount.name":"default","openebs.io/cas-type":"localpv-lvm","openebs.io/volgroup":"tank","storage.kubernetes.io/csiProvisionerIdentity":"1651419856903-8081-local.csi.openebs.io"},"volume_id":"pvc-81fb10f6-8b34-408f-a3ab-38faeb9c662a"}
Mon, May 2 2022 9:42:56 am | I0502 07:42:56.871463 1 mount_linux.go:366] Disk "/dev/tank/pvc-81fb10f6-8b34-408f-a3ab-38faeb9c662a" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/tank/pvc-81fb10f6-8b34-408f-a3ab-38faeb9c662a]
Mon, May 2 2022 9:42:57 am | I0502 07:42:57.321866 1 mount_linux.go:376] Disk successfully formatted (mkfs): ext4 - /dev/tank/pvc-81fb10f6-8b34-408f-a3ab-38faeb9c662a /var/lib/kubelet/pods/48d32d60-ca98-4861-83af-e1087eeb4673/volumes/kubernetes.io~csi/pvc-81fb10f6-8b34-408f-a3ab-38faeb9c662a/mount
Mon, May 2 2022 9:42:57 am | I0502 07:42:57.329406 1 mount.go:208] lvm: volume tank/pvc-81fb10f6-8b34-408f-a3ab-38faeb9c662a mounted /var/lib/kubelet/pods/48d32d60-ca98-4861-83af-e1087eeb4673/volumes/kubernetes.io~csi/pvc-81fb10f6-8b34-408f-a3ab-38faeb9c662a/mount fs ext4
Mon, May 2 2022 9:42:57 am | I0502 07:42:57.329442 1 grpc.go:81] GRPC response: {}
Mon, May 2 2022 9:43:13 am | I0502 07:43:13.278338 1 lvmnode.go:100] lvm node controller: node volume groups updated current=[{Name:tank UUID:vK4U2G-XIVd-naJl-eL88-YfIk-3rkj-TQ0BKA Size:{i:{value:429492535296 scale:0} d:{Dec:<nil>} s:409596Mi Format:BinarySI} Free:{i:{value:402124701696 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:3 PVCount:1}], required=[{Name:tank UUID:vK4U2G-XIVd-naJl-eL88-YfIk-3rkj-TQ0BKA Size:{i:{value:429492535296 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:396755992576 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:4 PVCount:1}]
Mon, May 2 2022 9:43:13 am | I0502 07:43:13.278661 1 lvmnode.go:110] lvm node controller: updating node object with &{TypeMeta:{Kind: APIVersion:} ObjectMeta:{Name:k8s-st-node GenerateName: Namespace:default SelfLink: UID:e5090f45-59a9-4474-8287-00c347ccc0de ResourceVersion:5406374 Generation:22 CreationTimestamp:2022-04-18 09:41:49 +0000 UTC DeletionTimestamp:<nil> DeletionGracePeriodSeconds:<nil> Labels:map[] Annotations:map[] OwnerReferences:[{APIVersion:v1 Kind:Node Name:k8s-st-node UID:4af5b9cf-94fd-4e4f-a5b7-24c37a91fcaf Controller:0xc0004878b8 BlockOwnerDeletion:<nil>}] Finalizers:[] ClusterName: ManagedFields:[{Manager:lvm-driver Operation:Update APIVersion:local.openebs.io/v1alpha1 Time:2022-04-18 09:41:49 +0000 UTC FieldsType:FieldsV1 FieldsV1:{"f:metadata":{"f:ownerReferences":{".":{},"k:{\"uid\":\"4af5b9cf-94fd-4e4f-a5b7-24c37a91fcaf\"}":{}}},"f:volumeGroups":{}}}]} VolumeGroups:[{Name:tank UUID:vK4U2G-XIVd-naJl-eL88-YfIk-3rkj-TQ0BKA Size:{i:{value:429492535296 scale:0} d:{Dec:<nil>} s: Format:BinarySI} Free:{i:{value:396755992576 scale:0} d:{Dec:<nil>} s: Format:BinarySI} LVCount:4 PVCount:1}]}
Mon, May 2 2022 9:43:13 am | I0502 07:43:13.284490 1 lvmnode.go:139] Got update event for lvm node default/k8s-st-node
Mon, May 2 2022 9:43:13 am | I0502 07:43:13.284436 1 lvmnode.go:114] lvm node controller: updated node object default/k8s-st-node
Mon, May 2 2022 9:43:13 am | I0502 07:43:13.284519 1 lvmnode.go:274] Successfully synced 'default/k8s-st-node'
Mon, May 2 2022 9:43:13 am | I0502 07:43:13.342360 1 lvmnode.go:274] Successfully synced 'default/k8s-st-node'
Mon, May 2 2022 9:44:13 am | I0502 07:44:13.294371 1 lvmnode.go:274] Successfully synced 'default/k8s-st-node'
Mon, May 2 2022 9:45:13 am | I0502 07:45:13.294421 1 lvmnode.go:274] Successfully synced 'default/k8s-st-node'
Mon, May 2 2022 9:46:13 am | I0502 07:46:13.298180 1 lvmnode.go:274] Successfully synced 'default/k8s-st-node'

controller:

Mon, May 2 2022 9:49:12 am | E0502 07:49:12.963643 1 volume.go:270] Get snapshot failed err: lvmsnapshots.local.openebs.io "snapshot-d57eaca6-477b-47e8-8a7e-2618f5bcdfd6" not found
Mon, May 2 2022 9:49:12 am | E0502 07:49:12.966607 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to handle CreateSnapshotRequest for pvc-81fb10f6-8b34-408f-a3ab-38faeb9c662a: snapshot-d57eaca6-477b-47e8-8a7e-2618f5bcdfd6, {LVMSnapshot.local.openebs.io "snapshot-d57eaca6-477b-47e8-8a7e-2618f5bcdfd6" is invalid: [spec.capacity: Required value, spec.vgPattern: Required value]}
Mon, May 2 2022 9:49:13 am | I0502 07:49:13.561757 1 grpc.go:72] GRPC call: /csi.v1.Identity/GetPluginInfo requests {}
Mon, May 2 2022 9:49:13 am | I0502 07:49:13.561814 1 grpc.go:81] GRPC response: {"name":"local.csi.openebs.io","vendor_version":"0.7.0"}
Mon, May 2 2022 9:49:13 am | I0502 07:49:13.562166 1 grpc.go:72] GRPC call: /csi.v1.Controller/CreateSnapshot requests {"name":"snapshot-d57eaca6-477b-47e8-8a7e-2618f5bcdfd6","parameters":{"snapSize":"50%"},"source_volume_id":"pvc-81fb10f6-8b34-408f-a3ab-38faeb9c662a"}
Mon, May 2 2022 9:49:13 am | I0502 07:49:13.562223 1 controller.go:566] CreateSnapshot volume snapshot-d57eaca6-477b-47e8-8a7e-2618f5bcdfd6 for pvc-81fb10f6-8b34-408f-a3ab-38faeb9c662a
Mon, May 2 2022 9:49:13 am | E0502 07:49:13.565636 1 volume.go:270] Get snapshot failed err: lvmsnapshots.local.openebs.io "snapshot-d57eaca6-477b-47e8-8a7e-2618f5bcdfd6" not found
Mon, May 2 2022 9:49:13 am | E0502 07:49:13.568670 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to handle CreateSnapshotRequest for pvc-81fb10f6-8b34-408f-a3ab-38faeb9c662a: snapshot-d57eaca6-477b-47e8-8a7e-2618f5bcdfd6, {LVMSnapshot.local.openebs.io "snapshot-d57eaca6-477b-47e8-8a7e-2618f5bcdfd6" is invalid: [spec.capacity: Required value, spec.vgPattern: Required value]}
Mon, May 2 2022 9:49:13 am | I0502 07:49:13.760866 1 grpc.go:72] GRPC call: /csi.v1.Identity/GetPluginInfo requests {}
Mon, May 2 2022 9:49:13 am | I0502 07:49:13.760915 1 grpc.go:81] GRPC response: {"name":"local.csi.openebs.io","vendor_version":"0.7.0"}
Mon, May 2 2022 9:49:13 am | I0502 07:49:13.761298 1 grpc.go:72] GRPC call: /csi.v1.Controller/CreateSnapshot requests {"name":"snapshot-d57eaca6-477b-47e8-8a7e-2618f5bcdfd6","parameters":{"snapSize":"50%"},"source_volume_id":"pvc-81fb10f6-8b34-408f-a3ab-38faeb9c662a"}
Mon, May 2 2022 9:49:13 am | I0502 07:49:13.761356 1 controller.go:566] CreateSnapshot volume snapshot-d57eaca6-477b-47e8-8a7e-2618f5bcdfd6 for pvc-81fb10f6-8b34-408f-a3ab-38faeb9c662a
Mon, May 2 2022 9:49:13 am | E0502 07:49:13.764340 1 volume.go:270] Get snapshot failed err: lvmsnapshots.local.openebs.io "snapshot-d57eaca6-477b-47e8-8a7e-2618f5bcdfd6" not found
Mon, May 2 2022 9:49:13 am | E0502 07:49:13.767479 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to handle CreateSnapshotRequest for pvc-81fb10f6-8b34-408f-a3ab-38faeb9c662a: snapshot-d57eaca6-477b-47e8-8a7e-2618f5bcdfd6, {LVMSnapshot.local.openebs.io "snapshot-d57eaca6-477b-47e8-8a7e-2618f5bcdfd6" is invalid: [spec.capacity: Required value, spec.vgPattern: Required value]}
Mon, May 2 2022 9:49:14 am | I0502 07:49:14.161135 1 grpc.go:72] GRPC call: /csi.v1.Identity/GetPluginInfo requests {}
Mon, May 2 2022 9:49:14 am | I0502 07:49:14.161183 1 grpc.go:81] GRPC response: {"name":"local.csi.openebs.io","vendor_version":"0.7.0"}
Mon, May 2 2022 9:49:14 am | I0502 07:49:14.161545 1 grpc.go:72] GRPC call: /csi.v1.Controller/CreateSnapshot requests {"name":"snapshot-d57eaca6-477b-47e8-8a7e-2618f5bcdfd6","parameters":{"snapSize":"50%"},"source_volume_id":"pvc-81fb10f6-8b34-408f-a3ab-38faeb9c662a"}
Mon, May 2 2022 9:49:14 am | I0502 07:49:14.161600 1 controller.go:566] CreateSnapshot volume snapshot-d57eaca6-477b-47e8-8a7e-2618f5bcdfd6 for pvc-81fb10f6-8b34-408f-a3ab-38faeb9c662a
Mon, May 2 2022 9:49:14 am | E0502 07:49:14.165042 1 volume.go:270] Get snapshot failed err: lvmsnapshots.local.openebs.io "snapshot-d57eaca6-477b-47e8-8a7e-2618f5bcdfd6" not found
Mon, May 2 2022 9:49:14 am | E0502 07:49:14.168257 1 grpc.go:79] GRPC error: rpc error: code = Internal desc = failed to handle CreateSnapshotRequest for pvc-81fb10f6-8b34-408f-a3ab-38faeb9c662a: snapshot-d57eaca6-477b-47e8-8a7e-2618f5bcdfd6, {LVMSnapshot.local.openebs.io "snapshot-d57eaca6-477b-47e8-8a7e-2618f5bcdfd6" is invalid: [spec.capacity: Required value, spec.vgPattern: Required value]}

Log in the controller repeats rapidly, it keeps trying to create the snapshot. No errors in daemonset at all

pawanpraka1 commented 2 years ago

What is the k8s and LVM-LocalPV version you are using?

pawanpraka1 commented 2 years ago

It seems like CRDs are not updated in your case. How did you install it?

benn0r commented 2 years ago

What is the k8s and LVM-LocalPV version you are using?

openebs/lvm-driver:0.7.0 -> is that what you asked? Kubernetes Version: v1.22.7 (K3s) -> installed with Rancher on Photon OS

It seems like CRDs are not updated in your case. How did you install it?

I installed openebs with the Partner Chart from Rancher: https://github.com/rancher/partner-charts/tree/main-source/charts/openebs/openebs/3.0.0

I can see now that openebs/lvm-driver:0.9.0 is already relased. Should i try to upgrade?

pawanpraka1 commented 2 years ago

@benn0r can you install 0.9.0 and try? Here is the helm chart location https://github.com/openebs/lvm-localpv/tree/develop/deploy/helm/charts.

pawanpraka1 commented 2 years ago

The CRDs here https://github.com/rancher/partner-charts/tree/main-source/charts/openebs/openebs/3.0.0 are outdated. Please install it from the LVM-LocalPV repo.

benn0r commented 2 years ago

The CRDs here https://github.com/rancher/partner-charts/tree/main-source/charts/openebs/openebs/3.0.0 are outdated. Please install it from the LVM-LocalPV repo.

You are right, thanks a lot. WIth openebs 3.2.0 chart and updating the crds it works now!