Closed pbs-jyu closed 1 month ago
i searched and found this mentioned globalmount: vsphere-csi-driver/pkg/csi/service/osutils/linux_os_utils.go
kubectl get CSINode NAME DRIVERS AGE s-mick8s1 1 18d s-mick8s2 1 18d s-mick8s3 1 18d
i saw this issue: https://github.com/kubernetes-sigs/vsphere-csi-driver/issues/124. The issue they had was related to CSINode not created. My CSI nodes are running fine. The similarit between their case and my issue are both has some error messages related to "globalmount not pre-created" it looks like "replacing all /var/lib/kubelet with /opt/rke/var/lib/kubelet" fixed the issue for two people for issue#124. vsphere-csi-driver.txt I have found "/var/lib/kubelet" in 6 places in vsphere-csi-driver.yaml (CSI V3.3). So for my case, should I replace all 6 place of "/var/lib/kubelet" with "/var/snap/microk8s/common/var/lib/kubelet"?
My errror message is ""/var/snap/microk8s/common/var/lib/kubelet/plugins/kubernetes.io/csi/csi.vsphere.vmware.com/xxxx/globalmount not pre-created""
Line 459: value: /var/lib/kubelet/plugins/csi.vsphere.vmware.com/csi.sock
Line 469: - --kubelet-registration-path=/var/lib/kubelet/plugins/csi.vsphere.vmware.com/csi.sock
Line 510: mountPath: /var/lib/kubelet
Line 543: path: /var/lib/kubelet/plugins_registry
Line 547: path: /var/lib/kubelet/plugins/csi.vsphere.vmware.com
Line 551: path: /var/lib/kubelet
"/var/lib/kubelet" has been found in 6 places in vsphere-csi-driver.yaml (CSI V3.3)
Line 459: value: /var/lib/kubelet/plugins/csi.vsphere.vmware.com/csi.sock
Line 469: - --kubelet-registration-path=/var/lib/kubelet/plugins/csi.vsphere.vmware.com/csi.sock
Line 510: mountPath: /var/lib/kubelet
Line 543: path: /var/lib/kubelet/plugins_registry
Line 547: path: /var/lib/kubelet/plugins/csi.vsphere.vmware.com
Line 551: path: /var/lib/kubelet
Line 459: value: /var/lib/kubelet/plugins/csi.vsphere.vmware.com/csi.sock
Line 469: - --kubelet-registration-path=/var/lib/kubelet/plugins/csi.vsphere.vmware.com/csi.sock
containers:
- name: node-driver-registrar
image: registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.10.1
env:
- name: ADDRESS
value: /csi/csi.sock
- name: DRIVER_REG_SOCK_PATH
value: /var/lib/kubelet/plugins/csi.vsphere.vmware.com/csi.sock # Line 459
volumeMounts:
- name: plugin-dir
mountPath: /csi
- name: registration-dir
mountPath: /registration
livenessProbe:
exec:
command:
- /csi-node-driver-registrar
- --kubelet-registration-path=/var/lib/kubelet/plugins/csi.vsphere.vmware.com/csi.sock # Line 469
Line 510: mountPath: /var/lib/kubelet
- name: vsphere-csi-node
image: gcr.io/cloud-provider-vsphere/csi/release/driver:v3.3.0
volumeMounts:
- name: plugin-dir
mountPath: /csi
- name: pods-mount-dir
mountPath: /var/lib/kubelet # line 510
# needed so that any mounts setup inside this container are
# propagated back to the host machine.
mountPropagation: "Bidirectional"
- name: device-dir
mountPath: /dev
- name: blocks-dir
mountPath: /sys/block
- name: sys-devices-dir
mountPath: /sys/devices
Line 543: path: /var/lib/kubelet/plugins_registry
Line 547: path: /var/lib/kubelet/plugins/csi.vsphere.vmware.com
Line 551: path: /var/lib/kubelet
volumes:
- name: registration-dir
hostPath:
path: /var/lib/kubelet/plugins_registry Line 543
type: Directory
- name: plugin-dir
hostPath:
path: /var/lib/kubelet/plugins/csi.vsphere.vmware.com Line 547
type: DirectoryOrCreate
- name: pods-mount-dir
hostPath:
path: /var/lib/kubelet Line 551
type: Directory
- name: device-dir
hostPath:
path: /dev
- name: blocks-dir
hostPath:
path: /sys/block
type: Directory
- name: sys-devices-dir
hostPath:
path: /sys/devices
type: Directory
tolerations:
- effect: NoExecute
operator: Exists
- effect: NoSchedule
operator: Exists
I have executed my thought during the weekend and it worked.
case is closed.
Is this a BUG REPORT or FEATURE REQUEST?: BUG REPORT
What happened:
Deployed VMware CPI and CSI. everything looks good to me. created SC, PVC. PVC ,PV created and look good. Saw the PV in vcenter under Container Volumes section. Deployed simple pod to use the PVC.
kg pods -n devops NAME READY STATUS RESTARTS AGE pod-w-persistent-volume 0/1 ContainerCreating 0 35m
k describe pods pod-w-persistent-volume -n devops Warning FailedMount 110s (x25 over 36m) kubelet MountVolume.MountDevice failed for volume "pvc-39ba841f-98a1-444f-8c8c-99b8ef408af4" : rpc error: code = FailedPrecondition desc = target: /var/snap/microk8s/common/var/lib/kubelet/plugins/kubernetes.io/csi/csi.vsphere.vmware.com/2d1572c99aa1abb2ae84760b5b45b331f97fbedcbb046e13a82c544ef5e7f727/globalmount not pre-created
k logs pod-w-persistent-volume -n devops Error from server (BadRequest): container "my-container" in pod "pod-w-persistent-volume" is waiting to start: ContainerCreating
What you expected to happen: The pod should be created with the volume is attached to /data
apiVersion: v1 kind: Pod metadata: name: pod-w-persistent-volume namespace: devops spec: containers:
How to reproduce it (as minimally and precisely as possible):
Anything else we need to know?: My Kubernetes cluster is Microk8s
Environment:
uname -a
): 122-Ubuntu SMP Thu May 23 07:48:21 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux