Cloudstack CSI Driver cannot mount pvc after attach volume on VMware

tungntvn23 commented 1 year ago

ISSUE TYPE

Bug Report

COMPONENT NAME

Storage csi driver, Kubernetes Service

CLOUDSTACK VERSION
4.17.2.0

CONFIGURATION

Advanced networking with vmware 7
Kubernetes service enabled
Cloudstack CSI driver deployed

OS / ENVIRONMENT

CentOS 7

SUMMARY

When create kubernetes persistent volume claim, cloudstack create volume and attach to node success. But when node find new disk and mount, error happen and cannot mount new disk.

STEPS TO REPRODUCE

Install the CSI driver in the K8s cluster using the command : kubectl apply -f https://github.com/apalia/cloudstack-csi-driver/releases/latest/download/manifest.yaml

Deploy this storage class:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: cloudstack-custom
provisioner: csi.cloudstack.apache.org
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
allowVolumeExpansion: false
parameters:
csi.cloudstack.apache.org/disk-offering-id: <id of custom disk offering>

Deploy this pvs and pod:

kind: PersistentVolumeClaim
metadata:
name: example-pvc
spec:
storageClassName: cloudstack-custom
accessModes:
- ReadWriteOnce
resources:
requests:
  storage: 1Gi
---
apiVersion: v1
kind: Pod
metadata:
name: example-pod
spec:
containers:
- name: example
  image: busybox
  volumeMounts:
    - mountPath: "/data"
      name: example-volume
  stdin: true
  stdinOnce: true
  tty: true
volumes:
- name: example-volume
  persistentVolumeClaim:
    claimName: example-pvc

EXPECTED RESULTS

PVC create, attach and mount successfull

ACTUAL RESULTS

PVC cannot mount on node. Cluster logs:


2023-05-11T14:53:12.66819279Z stderr F {"level":"warn","ts":1683816792.6673605,"caller":"mount/mount.go:110","msg":"Failed to rescan scsi host /sys/class/scsi_host/host0/scan","grpc.start_time":"2023-05-11T14:52:44Z","grpc.request.deadline":"2023-05-11T14:54:44Z","system":"grpc","span.kind":"server","grpc.service":"csi.v1.Node","grpc.method":"NodeStageVolume"}
2023-05-11T14:53:12.668231033Z stderr F {"level":"warn","ts":1683816792.6674378,"caller":"mount/mount.go:110","msg":"Failed to rescan scsi host /sys/class/scsi_host/host1/scan","grpc.start_time":"2023-05-11T14:52:44Z","grpc.request.deadline":"2023-05-11T14:54:44Z","system":"grpc","span.kind":"server","grpc.service":"csi.v1.Node","grpc.method":"NodeStageVolume"}
2023-05-11T14:53:12.668293633Z stderr F {"level":"warn","ts":1683816792.6674924,"caller":"mount/mount.go:110","msg":"Failed to rescan scsi host /sys/class/scsi_host/host2/scan","grpc.start_time":"2023-05-11T14:52:44Z","grpc.request.deadline":"2023-05-11T14:54:44Z","system":"grpc","span.kind":"server","grpc.service":"csi.v1.Node","grpc.method":"NodeStageVolume"}
2023-05-11T14:53:12.668320007Z stderr F {"level":"warn","ts":1683816792.667541,"caller":"mount/mount.go:110","msg":"Failed to rescan scsi host /sys/class/scsi_host/host3/scan","grpc.start_time":"2023-05-11T14:52:44Z","grpc.request.deadline":"2023-05-11T14:54:44Z","system":"grpc","span.kind":"server","grpc.service":"csi.v1.Node","grpc.method":"NodeStageVolume"}
2023-05-11T14:53:12.668342763Z stderr F {"level":"warn","ts":1683816792.6675792,"caller":"mount/mount.go:110","msg":"Failed to rescan scsi host /sys/class/scsi_host/host4/scan","grpc.start_time":"2023-05-11T14:52:44Z","grpc.request.deadline":"2023-05-11T14:54:44Z","system":"grpc","span.kind":"server","grpc.service":"csi.v1.Node","grpc.method":"NodeStageVolume"}
2023-05-11T14:53:12.668357687Z stderr F {"level":"warn","ts":1683816792.667621,"caller":"mount/mount.go:110","msg":"Failed to rescan scsi host /sys/class/scsi_host/host5/scan","grpc.start_time":"2023-05-11T14:52:44Z","grpc.request.deadline":"2023-05-11T14:54:44Z","system":"grpc","span.kind":"server","grpc.service":"csi.v1.Node","grpc.method":"NodeStageVolume"}
2023-05-11T14:53:12.668386373Z stderr F {"level":"warn","ts":1683816792.6677256,"caller":"mount/mount.go:121","msg":"Error running udevadm trigger executable file not found in $PATH\n","grpc.start_time":"2023-05-11T14:52:44Z","grpc.request.deadline":"2023-05-11T14:54:44Z","system":"grpc","span.kind":"server","grpc.service":"csi.v1.Node","grpc.method":"NodeStageVolume"}```

weizhouapache commented 1 year ago

@tungntvn23 what's the guest OS ?

we have noticed some difference between OSes, for example https://github.com/apache/cloudstack/issues/7316

tungntvn23 commented 1 year ago

@weizhouapache I use default system vm template for k8s node Debian GNU/Linux 11 (bullseye) and centos 7 for management with vmware 7.

tungntvn23 commented 1 year ago

i try execute command inside container and get error:

root@democluster04-node-18809e60f05:~# ctr --namespace=k8s.io task exec -t --exec-id bash1 6e17361f0d4e773da13ab3a32c24394a1fd4186198032d0c03a918b77afacb55 sh
# echo '- - -' > /sys/class/scsi_host/host0/scan
sh: can't create /sys/class/scsi_host/host0/scan: Read-only file system
# whoami
root
#

I also check privileged is enable.

zap51 commented 11 months ago

Hi, @kiranchavala @weizhouapache An official CSI for CloudStack would be really good. I've had experience with https://github.com/apalia/cloudstack-csi-driver, looks no longer maintained and doesn't work like with detaching volumes from the nodes, working with projects etc., Please let us know if we have a roadmap for this, this would be really appreciated :)

weizhouapache commented 10 months ago

moved to 4.18.2.0 milestone

apache / cloudstack