[BUG] - unix /var/lib/kubelet/plugins/block.csi.vultr.com/csi.sock not accessible (for rook.io)

[defaultbranch]

Describe the bug Using rook.io, the rook pods rook-ceph-osd-prepare- fails to setup a PersistentVolumeClaim.

"describe pod" finally reports the event (warning) "MapVolume.SetUpDevice failed for volume "pvc-c869a0057b0c4904" : kubernetes.io/csi: blockMapper.stageVolumeForBlock failed to check STAGE_UNSTAGE_VOLUME capability: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial unix /var/lib/kubelet/plugins/block.csi.vultr.com/csi.sock: connect: connection refused"" from the kubelet.

To Reproduce Steps to reproduce the behavior: (NOTE: This setup works fine on Azure AKS, with only the storageClassName adjusted.)

1. Create a fresh kubernetes cluster (probably 1 worker node is sufficient for reproduction)

2. For basic Rook setup: From the files in https://github.com/rook/rook/tree/master/deploy/examples, run kubectl apply -f for crds.yaml, common.yaml and operator.yaml, this creates CRDs, Roles and a rook-ceph-operator deployment/pod

3. Run kubectl apply -f for the following CephCluster yaml:

   apiVersion: ceph.rook.io/v1
   kind: CephCluster
   metadata:
   name: rook-ceph
   namespace: rook-ceph
   spec:
   cephVersion:
   # NOTE: see cluster.yaml in <https://github.com/rook/rook.git> for up-to-date image version 
   image: quay.io/ceph/ceph:v17.2.1
   allowUnsupported: false
   dataDirHostPath: /var/lib/rook
   skipUpgradeChecks: false
   continueUpgradeAfterChecksEvenIfNotHealthy: false
   waitTimeoutForHealthyOSDInMinutes: 10
   mon:
   count: 3
   allowMultiplePerNode: false
   mgr:
   count: 2
   allowMultiplePerNode: false
   modules:
     - name: pg_autoscaler
       enabled: true
   dashboard:
   enabled: true
   ssl: true
   storage:
   storageClassDeviceSets:
   - name: set1
     # NOTE: change this to the number of nodes that should host an OSD
     count: 1
     portable: false
     tuneDeviceClass: false
     encrypted: false
     volumeClaimTemplates:
     - metadata:
         name: data
       spec:
         storageClassName: vultr-block-storage-hdd
         accessModes:
           - ReadWriteOnce
         # NOTE: rook seems to expect a raw, unmounted device "volumeMode: Block"
         volumeMode: Block
         resources:
           requests:
             storage: 40Gi

4. See events in kubectl -n rook-ceph describe pod rook-ceph-osd-prepare-* after the pod(s) got stuck.

Expected behavior The pods rook-ceph-osd-prepare-* should disappear after a short time, and instead corresponding rook-ceph-osd-* pods (without -prepare-) should remain.

Additional context I was using VKE with Kubernetes 1.23.x.

[ddymko]

@defaultbranch

NOTE: rook seems to expect a raw, unmounted device "volumeMode: Block"

volumeMode: Block

IIRC the vultr-csi doesn't support a raw unmounted device

https://github.com/vultr/vultr-csi/blob/master/driver/mounter.go#L113

[kaznak]

Hi, I have encountered the same situation. I have a question about this matter.

1. do you plan to support volumeMode: Block in the future?
2. is it possible to support volumeMode: Block by changing this CSI implementation?

Thank you in advance.

[cuppett]

I have a question about this matter.

1. do you plan to support volumeMode: Block in the future?
2. is it possible to support volumeMode: Block by changing this CSI implementation?

Unofficially: I'd expect this should be possible (with changes). When attaching block devices to vultr instances normally, you get the raw device and can do stuff to them (create LVM volumes, basic filesystems, use LUKS, etc).