search

kubernetes-sigs / vsphere-csi-driver

vSphere storage Container Storage Interface (CSI) plugin
https://docs.vmware.com/en/VMware-vSphere-Container-Storage-Plug-in/index.html
Apache License 2.0
293 stars 177 forks source link

globalmount not pre-created #2952

Closed pbs-jyu closed 1 month ago

pbs-jyu commented 1 month ago

Is this a BUG REPORT or FEATURE REQUEST?: BUG REPORT

Uncomment only one, leave it on its own line:

kind bug /kind feature

What happened:
Deployed VMware CPI and CSI. everything looks good to me. created SC, PVC. PVC ,PV created and look good. Saw the PV in vcenter under Container Volumes section. Deployed simple pod to use the PVC.
kg pods -n devops NAME READY STATUS RESTARTS AGE pod-w-persistent-volume 0/1 ContainerCreating 0 35m

k describe pods pod-w-persistent-volume -n devops Warning FailedMount 110s (x25 over 36m) kubelet MountVolume.MountDevice failed for volume "pvc-39ba841f-98a1-444f-8c8c-99b8ef408af4" : rpc error: code = FailedPrecondition desc = target: /var/snap/microk8s/common/var/lib/kubelet/plugins/kubernetes.io/csi/csi.vsphere.vmware.com/2d1572c99aa1abb2ae84760b5b45b331f97fbedcbb046e13a82c544ef5e7f727/globalmount not pre-created

k logs pod-w-persistent-volume -n devops Error from server (BadRequest): container "my-container" in pod "pod-w-persistent-volume" is waiting to start: ContainerCreating

What you expected to happen: The pod should be created with the volume is attached to /data

apiVersion: v1 kind: Pod metadata: name: pod-w-persistent-volume namespace: devops spec: containers:

How to reproduce it (as minimally and precisely as possible):

  1. I have tried delete the pod, delete pvc, pv, even the sc. retry it still the same. "/var/snap/microk8s/common/var/lib/kubelet/plugins/kubernetes.io/csi/csi.vsphere.vmware.com/xxxx/globalmount not pre-created"
  2. i have even tried manually created the directory manually on each worker nodes, mainly the last two direcotry were not there. But the directoried created won't stay. It will be disappear later by itself.

Anything else we need to know?: My Kubernetes cluster is Microk8s

Environment:

pbs-jyu commented 1 month ago

i searched and found this mentioned globalmount: vsphere-csi-driver/pkg/csi/service/osutils/linux_os_utils.go

pbs-jyu commented 1 month ago

kubectl get CSINode NAME DRIVERS AGE s-mick8s1 1 18d s-mick8s2 1 18d s-mick8s3 1 18d

pbs-jyu commented 1 month ago

i saw this issue: https://github.com/kubernetes-sigs/vsphere-csi-driver/issues/124. The issue they had was related to CSINode not created. My CSI nodes are running fine. The similarit between their case and my issue are both has some error messages related to "globalmount not pre-created" it looks like "replacing all /var/lib/kubelet with /opt/rke/var/lib/kubelet" fixed the issue for two people for issue#124. vsphere-csi-driver.txt I have found "/var/lib/kubelet" in 6 places in vsphere-csi-driver.yaml (CSI V3.3). So for my case, should I replace all 6 place of "/var/lib/kubelet" with "/var/snap/microk8s/common/var/lib/kubelet"?

My errror message is ""/var/snap/microk8s/common/var/lib/kubelet/plugins/kubernetes.io/csi/csi.vsphere.vmware.com/xxxx/globalmount not pre-created""

Line 459:               value: /var/lib/kubelet/plugins/csi.vsphere.vmware.com/csi.sock
Line 469:               - --kubelet-registration-path=/var/lib/kubelet/plugins/csi.vsphere.vmware.com/csi.sock
Line 510:               mountPath: /var/lib/kubelet
Line 543:             path: /var/lib/kubelet/plugins_registry
Line 547:             path: /var/lib/kubelet/plugins/csi.vsphere.vmware.com
Line 551:             path: /var/lib/kubelet
pbs-jyu commented 1 month ago

"/var/lib/kubelet" has been found in 6 places in vsphere-csi-driver.yaml (CSI V3.3)

Line 459:               value: /var/lib/kubelet/plugins/csi.vsphere.vmware.com/csi.sock
Line 469:               - --kubelet-registration-path=/var/lib/kubelet/plugins/csi.vsphere.vmware.com/csi.sock
Line 510:               mountPath: /var/lib/kubelet
Line 543:             path: /var/lib/kubelet/plugins_registry
Line 547:             path: /var/lib/kubelet/plugins/csi.vsphere.vmware.com
Line 551:             path: /var/lib/kubelet

Line 459:               value: /var/lib/kubelet/plugins/csi.vsphere.vmware.com/csi.sock
Line 469:               - --kubelet-registration-path=/var/lib/kubelet/plugins/csi.vsphere.vmware.com/csi.sock
      containers:
    - name: node-driver-registrar
      image: registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.10.1
      env:
        - name: ADDRESS
          value: /csi/csi.sock
        - name: DRIVER_REG_SOCK_PATH
          value: /var/lib/kubelet/plugins/csi.vsphere.vmware.com/csi.sock        # Line 459
      volumeMounts:
        - name: plugin-dir
          mountPath: /csi
        - name: registration-dir
          mountPath: /registration
      livenessProbe:
        exec:
          command:
          - /csi-node-driver-registrar
          - --kubelet-registration-path=/var/lib/kubelet/plugins/csi.vsphere.vmware.com/csi.sock    # Line 469

Line 510:               mountPath: /var/lib/kubelet
    - name: vsphere-csi-node
      image: gcr.io/cloud-provider-vsphere/csi/release/driver:v3.3.0

      volumeMounts:
        - name: plugin-dir
          mountPath: /csi
        - name: pods-mount-dir
          mountPath: /var/lib/kubelet                                              # line 510
          # needed so that any mounts setup inside this container are
          # propagated back to the host machine.
          mountPropagation: "Bidirectional"
        - name: device-dir
          mountPath: /dev
        - name: blocks-dir
          mountPath: /sys/block
        - name: sys-devices-dir
          mountPath: /sys/devices

Line 543:             path: /var/lib/kubelet/plugins_registry
Line 547:             path: /var/lib/kubelet/plugins/csi.vsphere.vmware.com
Line 551:             path: /var/lib/kubelet

          volumes:
            - name: registration-dir
              hostPath:
                path: /var/lib/kubelet/plugins_registry                      Line 543
                type: Directory
            - name: plugin-dir
              hostPath:
                path: /var/lib/kubelet/plugins/csi.vsphere.vmware.com         Line 547
                type: DirectoryOrCreate
            - name: pods-mount-dir
              hostPath:
                path: /var/lib/kubelet                                        Line 551
                type: Directory
            - name: device-dir
              hostPath:
                path: /dev
            - name: blocks-dir
              hostPath:
                path: /sys/block
                type: Directory
            - name: sys-devices-dir
              hostPath:
                path: /sys/devices
                type: Directory
          tolerations:
            - effect: NoExecute
              operator: Exists
            - effect: NoSchedule
              operator: Exists
pbs-jyu commented 1 month ago

I have executed my thought during the weekend and it worked.

pbs-jyu commented 1 month ago

case is closed.