Csi primera "is already mounted error"

[alexbarta]

Hi,

We are having issues running hpe-csi-driver for primera with FC and multipath enabled. We are running a kubernetes cluster v1.20.2 on a baremetal infra and we are trying to create volumes on a Primera FC storage version 4.3.x We have followed csi driver 2.0.0 deployment guide https://scod.hpedev.io/csi_driver/deployment.html and we chose helm as installation method. Currently the driver is able to create pvc- virtual volumes and export the volumes to hosts. The pvc attachment to pods fails with following error Warning FailedMount 72s kubelet MountVolume.MountDevice failed for volume "pvc-6877504c-4a3b-4c40-ab22-3ffc0751c29d" : rpc error: code = Internal desc = Failed to stage volume pvc-6877504c-4a3b-4c40-ab22-3ffc0751c29d, err: failed to mount device /dev/mapper/mpathh, Error mounting device /dev/mapper/mpathh on mountpoint /var/lib/kubelet/plugins/hpe.com/mounts/pvc-6877504c-4a3b-4c40-ab22-3ffc0751c29d with options [], unable to mount the device at mountPoint : /var/lib/kubelet/plugins/hpe.com/mounts/pvc-6877504c-4a3b-4c40-ab22-3ffc0751c29d. Error: device /dev/mapper/mpathh is not mounted at /var/lib/kubelet/plugins/hpe.com/mounts/pvc-6877504c-4a3b-4c40-ab22-3ffc0751c29d Warning FailedMount 33s (x6 over 70s) kubelet MountVolume.MountDevice failed for volume "pvc-6877504c-4a3b-4c40-ab22-3ffc0751c29d" : rpc error: code = Internal desc = Failed to stage volume pvc-6877504c-4a3b-4c40-ab22-3ffc0751c29d, err: failed to mount device /dev/mapper/mpathh, Error mounting device /dev/mapper/mpathh on mountpoint /var/lib/kubelet/plugins/hpe.com/mounts/pvc-6877504c-4a3b-4c40-ab22-3ffc0751c29d with options [], unable to mount the device at mountPoint : /var/lib/kubelet/plugins/hpe.com/mounts/pvc-6877504c-4a3b-4c40-ab22-3ffc0751c29d. Error: command mount failed with rc=32 err=mount: /dev/mapper/mpathh is already mounted or /var/lib/docker/kubelet/plugins/hpe.com/mounts/pvc-6877504c-4a3b-4c40-ab22-3ffc0751c29d busy /dev/mapper/mpathh is already mounted on /var/lib/docker/kubelet/plugins/hpe.com/mounts/pvc-6877504c-4a3b-4c40-ab22-3ffc0751c29d I can see the volume mounted correctly behind multipath config:

[root@hpecp12 ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 500G 0 disk ├─sda1 8:1 0 1G 0 part /boot └─sda2 8:2 0 499G 0 part ├─centos-root 253:0 0 50G 0 lvm / ├─centos-swap 253:1 0 4G 0 lvm [SWAP] ├─centos-srv 253:2 0 100G 0 lvm /srv ├─centos-opt 253:3 0 100G 0 lvm /opt └─centos-var 253:4 0 150G 0 lvm /var sdb 8:16 0 2G 0 disk └─mpathh 253:6 0 2G 0 mpath /var/lib/docker/kubelet/plugins/hpe.com/mounts/pvc-6877504c-4a3b-4c40-ab22-3f sdc 8:32 0 1.2T 0 disk sdd 8:48 0 2G 0 disk └─mpathh 253:6 0 2G 0 mpath /var/lib/docker/kubelet/plugins/hpe.com/mounts/pvc-6877504c-4a3b-4c40-ab22-3f sde 8:64 0 1.2T 0 disk └─sde1 8:65 0 1.2T 0 part └─VolBDSCStore-thinpool 253:5 0 1.2T 0 lvm /var/lib/docker sdf 8:80 0 2G 0 disk └─mpathh 253:6 0 2G 0 mpath /var/lib/docker/kubelet/plugins/hpe.com/mounts/pvc-6877504c-4a3b-4c40-ab22-3f sdg 8:96 0 2G 0 disk └─mpathh 253:6 0 2G 0 mpath /var/lib/docker/kubelet/plugins/hpe.com/mounts/pvc-6877504c-4a3b-4c40-ab22-3f loop0 7:0 0 1.2T 0 loop loop1 7:1 0 1.2T 0 loop loop3 7:3 0 1.2T 0 loop

Any thougths?

KR

[datamattsson]

Hi, I apologize the late response. Could you redeploy the chart with the --set kubeletRootDir=/var/lib/docker/kubelet parameter passed to helm and see if it works?

[DiegoDelgadoMex]

This did the trick! I got the pvc mounted to the Pods now!

Thanks!

[datamattsson]

Thanks for verifying @DiegoDelgadoMex! @alexbarta do you mind checking your CSI driver installation as well so we can close?

[alexbarta]

Thanks @datamattsson it works! Now it's clear what caused the issue. I think you may close this one.