[BUG] MountDevice failed while expanding volume

patrik-upspot commented 2 years ago

Describe the bug

After a while my pod dies and does not start again. I get the following events

MountVolume. for volume "pvc-6911563e-37e6-4e08-a857-11267216d408" : Expander.NodeExpand found CSI plugin kubernetes.io/csi/driver.longhorn.io to not support node expansion
Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[data kube-api-access-pxgzh]: timed out waiting for the condition
Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[kube-api-access-pxgzh data]: timed out waiting for the condition
MountVolume.MountDevice failed for volume "pvc-6911563e-37e6-4e08-a857-11267216d408" : rpc error: code = InvalidArgument desc = volume pvc-6911563e-37e6-4e08-a857-11267216d408 hasn't been attached yet
Successfully assigned production/document-management-api-mongodb-56fbbbd46b-rjxxc to 89.58.1.150

To Reproduce

-

Expected behavior

Stable Storages

Log or Support bundle

If applicable, add the Longhorn managers' log or support bundle when the issue happens. You can generate a Support Bundle using the link at the footer of the Longhorn UI.

Environment

Longhorn version: 1.2.3
Installation method (e.g. Rancher Catalog App/Helm/Kubectl): Kubectl
Kubernetes distro (e.g. RKE/K3s/EKS/OpenShift) and version: Server Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.7", GitCommit:"1f86634ff08f37e54e8bfcd86bc90b61c98f84d4", GitTreeState:"clean", BuildDate:"2021-11-17T14:35:38Z", GoVersion:"go1.16.10", Compiler:"gc", Platform:"linux/amd64"}
- Number of management node in the cluster: 1
- Number of worker node in the cluster: 2
Node config
- OS type and version:
- CPU per node: 4
- Memory per node: 12
- Disk type(e.g. SSD/NVMe): SSD
- Network bandwidth between the nodes: 500mbit vlan ( Hoster: netcup )
Underlying Infrastructure (e.g. on AWS/GCE, EKS/GKE, VMWare/KVM, Baremetal): KVM
Number of Longhorn volumes in the cluster: 16

patrik-upspot commented 2 years ago

I can not post the Support-Bundle i only get the message "is not included in the list" Where can i put it ?

jenting commented 2 years ago

I can not post the Support-Bundle i only get the message "is not included in the list" Where can i put it ?

Mail to longhorn-support-bundle@suse.com to see if it's possible.

patrik-upspot commented 2 years ago

I have sent you a mail with a link to download it. The zip is ~50MB, its too big for email.

jenting commented 2 years ago

Which tool do you use to achieve the support bundle? I can't unzip it.

patrik-upspot commented 2 years ago

I used 7zip for compression. I zipped the bundle again do reduce the size.

jenting commented 2 years ago

Note, we need to check the flow of CSI plugins with:

ControllerExpandVolume
NodeStageVolume
NodeExpandVolume

jenting commented 2 years ago

How did you perform the volume expansion? From the Longhorn UI or directly increase the PVC spec.resources.requests.storage size or ...?

Besides that, I can't find any log related to ControllerExpandVolume in the CSI plugins log.

patrik-upspot commented 2 years ago

That's the strange thing. I have done nothing of the sort.

jenting commented 2 years ago

That's the strange thing. I have done nothing of the sort.

But you mentioned that the MountDevice failed while expanding volume. So, you did not perform the volume expansion in the end?

patrik-upspot commented 2 years ago

Exactly, I received this error message. But I haven't done anything.

jenting commented 2 years ago

I see. Then is this happened one volume only?

patrik-upspot commented 2 years ago

Yes, only one Volume has this problem. Do you have any idea what i can do?

patrik-upspot commented 2 years ago

What can i do to get the volume/pod running?

gricuk commented 2 years ago

the same problem, is there any way to make the pod alive?

joshimoo commented 2 years ago

@gricuk please provide the complete context for your case including a support bundle :)

josmare commented 1 year ago

I have similar error:

Warning  FailedMount             2s (x8 over 66s)  kubelet                  MountVolume.MountDevice failed while expanding volume for volume "my-vl" : Expander.NodeExpand found CSI plugin [kubernetes.io/csi/driver.longhorn.io](http://kubernetes.io/csi/driver.longhorn.io) to not support node expansion

Yes, I expanded the volume successfully, but the node is unable to attach it.

My Longhorn version is: v1.2.2

srujankujmar commented 9 months ago

I have similar error

Warning FailedMount 2m58s (x25 over 4m34s) kubelet MountVolume.Setup failed while expanding volume for volume "yggdrasil-inbound-pv" : Expander.NodeExpand found CSI plugin kubernetes.io/csi/efs.csi.aws.com to not support node expansion

Could you please let us know if the issue is resolved or not

longhorn / longhorn