LVM node can't create new PV

[styshoo]

What steps did you take and what happened: [A clear and concise description of what the bug is, and what commands you ran.] The lvm csi runs correctly at first. But after a longtime running(maybe two months), I create a StatefulSet with lvm pvc, but the pv can't be created.

What did you expect to happen: PV can be successfully created.

The output of the following commands will help us better understand what's going on: (Pasting long output into a GitHub gist or other Pastebin is fine.) describe pod:

• kubectl get pods -n openebs

  # kubectl get pod -n openebs 
  NAME                                                 READY   STATUS    RESTARTS   AGE
  kg-lvm-csi-lvm-localpv-controller-8487d5745c-4ln7x   5/5     Running   0          5d
  kg-lvm-csi-lvm-localpv-node-54tmn                    2/2     Running   0          5d

• kubectl get lvmvol -A -o yaml

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.] In the csi node log, if a pv can be successfully created, the log should be as following:

I0910 07:42:51.015586       1 volume.go:85] Got add event for Vol pvc-24626e4e-6f29-4fbc-a59b-a4f9b51a6ae9
I0910 07:42:51.015605       1 volume.go:86] lvmvolume object to be enqueued by Add handler: 
I0910 07:42:51.015690       1 volume.go:55] Getting lvmvol object name:pvc-24626e4e-6f29-4fbc-a59b-a4f9b51a6ae9, ns from cache

However, in my environment, we only get the lines "Got add event for Vo" and "lvmvolume object to be enqueued by Add handler", without the third line. After I reboot the csi node pod, everything is fine again. I can create pv successfully.

So, I doubt that there is something wrong with the workqueue, LVMVolume is put into the workqueue, but in the processNextWorkItem loop, nothing can be get from the queue. But I'm not sure with that, is there a clue ?

Environment:

• LVM Driver version: 1.5.0
• Kubernetes version (use kubectl version):1.24.1
• Kubernetes installer & version:
• Cloud provider or hardware configuration:
• OS (e.g. from /etc/os-release):

[abhilashshetty04]

Hi @styshoo , Can you provide more details on the workflow..

kubectl get sc <sc-name> 
kubectl get pvc <pvc-name>
kubectl get lvmvolume <pv-name> -n openebs

Do you mean after restarting lvm-node plugin lv gets created successfully?