search

kubernetes-sigs / vsphere-csi-driver

vSphere storage Container Storage Interface (CSI) plugin
https://docs.vmware.com/en/VMware-vSphere-Container-Storage-Plug-in/index.html
Apache License 2.0
295 stars 177 forks source link

VMWare 7, vsphere-csi-driver v2.0.0 - PVC is in Pending state #548

Closed tmarkunin closed 3 years ago

tmarkunin commented 3 years ago

/kind bug

What happened:

Hello,

I've followed https://veducate.co.uk/how-to-install-vsphere-csi-driver-openshift/ to install CSI driver. VMWare admin user was used for testing purposes.

I've created new PVC and it's in Pending state.

What you expected to happen:

PV should be created and PVC should be bounded.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?: csi-provisioner container log contains: _Failed to provision volume with StorageClass "sc": rpc error: code = Internal desc = failed to get shared datastores in kubernetes cluster. Error: Empty List of Node VMs returned from nodeManager__

Environment:

RaunakShah commented 3 years ago

@tmarkunin have you installed CPI on your cluster? Can you verify if ProviderID is set on all your kubernetes nodes by running the command:

kubectl describe nodes | egrep "ProviderID:|Name:"
tmarkunin commented 3 years ago

@RaunakShah here is the output: [root@oct-tst openshift-config-file]# oc describe nodes | egrep "ProviderID:|Name:"

Name:               osd-bsd7j-master-0

ProviderID:                               vsphere://420be7de-3432-5e66-765e-eb46fb19abca

Name:               osd-bsd7j-master-1

ProviderID:                               vsphere://420befa7-0832-ead7-b2a9-58bc5fa044a0

Name:               osd-bsd7j-master-2

ProviderID:                                         vsphere://420b13e1-eb7e-84e6-8fee-d4c52c71b1f5

Name:               osd-bsd7j-worker-gd574

ProviderID:                                vsphere://420b0aa9-93de-ada5-b3d7-7d11fff475ee

Name:               osd-bsd7j-worker-pqd5m

ProviderID:                               vsphere://420b9a4b-02cf-34bb-8b23-603ba3ef3ff2

Name:               osd-bsd7j-worker-sntsj

ProviderID:                               vsphere://420b67c1-57ca-15f1-01c6-9332b543cc8a

RaunakShah commented 3 years ago

Thanks. And can you verify if disk.enableUUID is set on all the VMs?

RaunakShah commented 3 years ago

If it hasn't been set, you'd need to provide us with CSI controller logs to help debug. I'm fairly certain the issue is somewhere in the configuration.

tmarkunin commented 3 years ago

@RaunakShah Yes, disk.enableUUID is set on all VMs.

tmarkunin commented 3 years ago

@RaunakShah Should I provide any logs?

RaunakShah commented 3 years ago

@tmarkunin yes, can you provide csi controller logs? Also, could you paste the output of csi-vsphere.conf file? Feel free to remove any sensitive information, i'm checking if the parameters provided are correct..

tmarkunin commented 3 years ago

@RaunakShah Sorry, the clipboard is blocked on the remote station. I can only upload images at the moment.

image

image

Configfile is in the next comment

tmarkunin commented 3 years ago

@RaunakShah added config

image

chapai518 commented 3 years ago 
{"level":"info","time":"2020-12-15T13:01:47.341367872Z","caller":"logger/logger.go:37","msg":"Setting default log level to :\"PRODUCTION\""}
{"level":"info","time":"2020-12-15T13:01:47.341637903Z","caller":"config/config.go:272","msg":"No Net Permissions given in Config. Using default permissions.","TraceId":"604f695f-4f68-40a2-a8bd-bc38e58b3669"}
{"level":"info","time":"2020-12-15T13:01:47.341705167Z","caller":"vanilla/controller.go:94","msg":"Initializing CNS controller","TraceId":"6c024745-3e4e-474b-86e9-49a3864bb464"}
{"level":"info","time":"2020-12-15T13:01:47.341724442Z","caller":"vsphere/virtualcentermanager.go:64","msg":"Initializing defaultVirtualCenterManager...","TraceId":"6c024745-3e4e-474b-86e9-49a3864bb464"}
{"level":"info","time":"2020-12-15T13:01:47.341738204Z","caller":"vsphere/virtualcentermanager.go:66","msg":"Successfully initialized defaultVirtualCenterManager","TraceId":"6c024745-3e4e-474b-86e9-49a3864bb464"}
{"level":"info","time":"2020-12-15T13:01:47.341757254Z","caller":"vsphntmkere/virtualcentermanager.go:110","msg":"Successfully registered VC \"vim-002.domain.com\"","TraceId":"6c024745-3e4e-474b-86e9-49a3864bb464"}
{"level":"info","time":"2020-12-15T13:01:47.34177286Z","caller":"volume/manager.go:93","msg":"Initializing new volume.defaultManager...","TraceId":"6c024745-3e4e-474b-86e9-49a3864bb464"}
{"level":"info","time":"2020-12-15T13:01:47.429761888Z","caller":"vsphere/virtualcenter.go:143","msg":"New session ID for 'VSPHERE.LOCAL\\Administrator' = 5257b845-fa2a-1bba-7774-e7a427dbb5db","TraceId":"6c024745-3e4e-474b-86e9-49a3864bb464"}
{"level":"info","time":"2020-12-15T13:01:47.4298013Z","caller":"node/manager.go:75","msg":"Initializing node.defaultManager...","TraceId":"6c024745-3e4e-474b-86e9-49a3864bb464"}
{"level":"info","time":"2020-12-15T13:01:47.429830592Z","caller":"node/manager.go:79","msg":"node.defaultManager initialized","TraceId":"6c024745-3e4e-474b-86e9-49a3864bb464"}
{"level":"info","time":"2020-12-15T13:01:47.429837835Z","caller":"kubernetes/kubernetes.go:67","msg":"k8s client using in-cluster config","TraceId":"6c024745-3e4e-474b-86e9-49a3864bb464"}
{"level":"info","time":"2020-12-15T13:01:47.43142947Z","caller":"vanilla/controller.go:181","msg":"Adding watch on path: \"/etc/cloud\"","TraceId":"6c024745-3e4e-474b-86e9-49a3864bb464"}
{"level":"info","time":"2020-12-15T13:01:47.431475936Z","caller":"service/service.go:106","msg":"configured: \"csi.vsphere.vmware.com\" with clusterFlavor: \"VANILLA\" and mode: \"controller\"","TraceId":"604f695f-4f68-40a2-a8bd-bc38e58b3669"}
time="2020-12-15T13:01:47Z" level=info msg="identity service registered"
time="2020-12-15T13:01:47Z" level=info msg="controller service registered"
time="2020-12-15T13:01:47Z" level=info msg=serving endpoint="unix:///var/lib/csi/sockets/pluginproxy/csi.sock"
{"level":"info","time":"2020-12-15T13:01:47.442279621Z","caller":"node/manager.go:103","msg":"Successfully registered node: \"osd-bsd7j-master-1\" with nodeUUID \"420befa7-0832-ead7-b2a9-58bc5fa044a0\"","TraceId":"e0da4bf9-8bf4-4b32-838a-4690d4c77443"}
{"level":"info","time":"2020-12-15T13:01:47.442345877Z","caller":"vsphere/virtualmachine.go:125","msg":"Initiating asynchronous datacenter listing with uuid 420befa7-0832-ead7-b2a9-58bc5fa044a0","TraceId":"e0da4bf9-8bf4-4b32-838a-4690d4c77443"}
{"level":"error","time":"2020-12-15T13:01:47.447083828Z","caller":"vsphere/virtualcenter.go:284","msg":"failed to fetch datacenter given dcPath SC5020-06-OpenShift with err: datacenter 'SC5020-06-OpenShift' not found","TraceId":"e0da4bf9-8bf4-4b32-838a-4690d4c77443","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/vsphere.(*VirtualCenter).getDatacenters\n\t/build/pkg/common/cns-lib/vsphere/virtualcenter.go:284\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/vsphere.(*VirtualCenter).GetDatacenters\n\t/build/pkg/common/cns-lib/vsphere/virtualcenter.go:300\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/vsphere.asyncGetAllDatacenters\n\t/build/pkg/common/cns-lib/vsphere/datacenter.go:136"}
{"level":"error","time":"2020-12-15T13:01:47.447122972Z","caller":"vsphere/datacenter.go:138","msg":"failed to fetch datacenters for vc vim-002.domain.com with err: datacenter 'SC5020-06-OpenShift' not found","TraceId":"e0da4bf9-8bf4-4b32-838a-4690d4c77443","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/vsphere.asyncGetAllDatacenters\n\t/build/pkg/common/cns-lib/vsphere/datacenter.go:138"}
{"level":"error","time":"2020-12-15T13:01:47.447155452Z","caller":"vsphere/virtualmachine.go:149","msg":"AsyncGetAllDatacenters with uuid 420befa7-0832-ead7-b2a9-58bc5fa044a0 sent an error: datacenter 'SC5020-06-OpenShift' not found","TraceId":"e0da4bf9-8bf4-4b32-838a-4690d4c77443","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/vsphere.GetVirtualMachineByUUID.func1\n\t/build/pkg/common/cns-lib/vsphere/virtualmachine.go:149"}
{"level":"error","time":"2020-12-15T13:01:47.447178377Z","caller":"vsphere/virtualmachine.go:190","msg":"Returning err: datacenter 'SC5020-06-OpenShift' not found for UUID 420befa7-0832-ead7-b2a9-58bc5fa044a0","TraceId":"e0da4bf9-8bf4-4b32-838a-4690d4c77443","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/vsphere.GetVirtualMachineByUUID\n\t/build/pkg/common/cns-lib/vsphere/virtualmachine.go:190\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).DiscoverNode\n\t/build/pkg/common/cns-lib/node/manager.go:117\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).RegisterNode\n\t/build/pkg/common/cns-lib/node/manager.go:104\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).nodeAdd\n\t/build/pkg/csi/service/vanilla/nodes.go:63\nk8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd\n\t/go/pkg/mod/k8s.io/client-go@v11.0.1-0.20191029005444-8e4128053008+incompatible/tools/cache/controller.go:195\nk8s.io/client-go/tools/cache.(*processor...
{"level":"error","time":"2020-12-15T13:01:47.44722414Z","caller":"node/manager.go:119","msg":"Couldn't find VM instance with nodeUUID 420befa7-0832-ead7-b2a9-58bc5fa044a0, failed to discover with err: datacenter 'SC5020-06-OpenShift' not found","TraceId":"e0da4bf9-8bf4-4b32-838a-4690d4c77443","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).DiscoverNode\n\t/build/pkg/common/cns-lib/node/manager.go:119\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).RegisterNode\n\t/build/pkg/common/cns-lib/node/manager.go:104\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).nodeAdd\n\t/build/pkg/csi/service/vanilla/nodes.go:63\nk8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd\n\t/go/pkg/mod/k8s.io/client-go@v11.0.1-0.20191029005444-8e4128053008+incompatible/tools/cache/controller.go:195\nk8s.io/client-go/tools/cache.(*processorListener).run.func1.1\n\t/go/pkg/mod/k8s.io/client-go@v11.0.1-0.20191029005444-8e4128053008+incompatible/to...
{"level":"error","time":"2020-12-15T13:01:47.447257857Z","caller":"node/manager.go:106","msg":"failed to discover VM with uuid: \"420befa7-0832-ead7-b2a9-58bc5fa044a0\" for node: \"osd-bsd7j-master-1\"","TraceId":"e0da4bf9-8bf4-4b32-838a-4690d4c77443","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).RegisterNode\n\t/build/pkg/common/cns-lib/node/manager.go:106\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).nodeAdd\n\t/build/pkg/csi/service/vanilla/nodes.go:63\nk8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd\n\t/go/pkg/mod/k8s.io/client-go@v11.0.1-0.20191029005444-8e4128053008+incompatible/tools/cache/controller.go:195\nk8s.io/client-go/tools/cache.(*processorListener).run.func1.1\n\t/go/pkg/mod/k8s.io/client-go@v11.0.1-0.20191029005444-8e4128053008+incompatible/tools/cache/shared_informer.go:554\nk8s.io/apimachinery/pkg/util/wait.ExponentialBackoff\n\t/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20191004074956-c5d2f014d689/pkg/util/wait/wait.go...
{"level":"warn","time":"2020-12-15T13:01:47.447284509Z","caller":"vanilla/nodes.go:65","msg":"failed to register node:\"osd-bsd7j-master-1\". err=datacenter 'SC5020-06-OpenShift' not found","TraceId":"e0da4bf9-8bf4-4b32-838a-4690d4c77443"}
{"level":"info","time":"2020-12-15T13:01:47.447361658Z","caller":"node/manager.go:103","msg":"Successfully registered node: \"osd-bsd7j-master-2\" with nodeUUID \"420b13e1-eb7e-84e6-8fee-d4c52c71b1f5\"","TraceId":"8a4016b1-9623-4d5a-9f76-e078b74fca77"}
{"level":"info","time":"2020-12-15T13:01:47.447402001Z","caller":"vsphere/virtualmachine.go:125","msg":"Initiating asynchronous datacenter listing with uuid 420b13e1-eb7e-84e6-8fee-d4c52c71b1f5","TraceId":"8a4016b1-9623-4d5a-9f76-e078b74fca77"}
{"level":"error","time":"2020-12-15T13:01:47.453196523Z","caller":"vsphere/virtualcenter.go:284","msg":"failed to fetch datacenter given dcPath SC5020-06-OpenShift with err: datacenter 'SC5020-06-OpenShift' not found","TraceId":"8a4016b1-9623-4d5a-9f76-e078b74fca77","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/vsphere.(*VirtualCenter).getDatacenters\n\t/build/pkg/common/cns-lib/vsphere/virtualcenter.go:284\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/vsphere.(*VirtualCenter).GetDatacenters\n\t/build/pkg/common/cns-lib/vsphere/virtualcenter.go:300\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/vsphere.asyncGetAllDatacenters\n\t/build/pkg/common/cns-lib/vsphere/datacenter.go:136"}
{"level":"error","time":"2020-12-15T13:01:47.453276835Z","caller":"vsphere/datacenter.go:138","msg":"failed to fetch datacenters for vc vim-002.domain.com with err: datacenter 'SC5020-06-OpenShift' not found","TraceId":"8a4016b1-9623-4d5a-9f76-e078b74fca77","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/vsphere.asyncGetAllDatacenters\n\t/build/pkg/common/cns-lib/vsphere/datacenter.go:138"}
{"level":"error","time":"2020-12-15T13:01:47.453328105Z","caller":"vsphere/virtualmachine.go:149","msg":"AsyncGetAllDatacenters with uuid 420b13e1-eb7e-84e6-8fee-d4c52c71b1f5 sent an error: datacenter 'SC5020-06-OpenShift' not found","TraceId":"8a4016b1-9623-4d5a-9f76-e078b74fca77","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/vsphere.GetVirtualMachineByUUID.func1\n\t/build/pkg/common/cns-lib/vsphere/virtualmachine.go:149"}
{"level":"error","time":"2020-12-15T13:01:47.453359142Z","caller":"vsphere/virtualmachine.go:190","msg":"Returning err: datacenter 'SC5020-06-OpenShift' not found for UUID 420b13e1-eb7e-84e6-8fee-d4c52c71b1f5","TraceId":"8a4016b1-9623-4d5a-9f76-e078b74fca77","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/vsphere.GetVirtualMachineByUUID\n\t/build/pkg/common/cns-lib/vsphere/virtualmachine.go:190\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).DiscoverNode\n\t/build/pkg/common/cns-lib/node/manager.go:117\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).RegisterNode\n\t/build/pkg/common/cns-lib/node/manager.go:104\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).nodeAdd\n\t/build/pkg/csi/service/vanilla/nodes.go:63\nk8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd\n\t/go/pkg/mod/k8s.io/client-go@v11.0.1-0.20191029005444-8e4128053008+incompatible/tools/cache/controller.go:195\nk8s.io/client-go/tools/cache.(*processor...
{"level":"error","time":"2020-12-15T13:01:47.453437275Z","caller":"node/manager.go:119","msg":"Couldn't find VM instance with nodeUUID 420b13e1-eb7e-84e6-8fee-d4c52c71b1f5, failed to discover with err: datacenter 'SC5020-06-OpenShift' not found","TraceId":"8a4016b1-9623-4d5a-9f76-e078b74fca77","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).DiscoverNode\n\t/build/pkg/common/cns-lib/node/manager.go:119\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).RegisterNode\n\t/build/pkg/common/cns-lib/node/manager.go:104\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).nodeAdd\n\t/build/pkg/csi/service/vanilla/nodes.go:63\nk8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd\n\t/go/pkg/mod/k8s.io/client-go@v11.0.1-0.20191029005444-8e4128053008+incompatible/tools/cache/controller.go:195\nk8s.io/client-go/tools/cache.(*processorListener).run.func1.1\n\t/go/pkg/mod/k8s.io/client-go@v11.0.1-0.20191029005444-8e4128053008+incompatible/t...
{"level":"error","time":"2020-12-15T13:01:47.453498683Z","caller":"node/manager.go:106","msg":"failed to discover VM with uuid: \"420b13e1-eb7e-84e6-8fee-d4c52c71b1f5\" for node: \"osd-bsd7j-master-2\"","TraceId":"8a4016b1-9623-4d5a-9f76-e078b74fca77","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/node.(*defaultManager).RegisterNode\n\t/build/pkg/common/cns-lib/node/manager.go:106\nsigs.k8s.io/vsphere-csi-driver/pkg/csi/service/vanilla.(*Nodes).nodeAdd\n\t/build/pkg/csi/service/vanilla/nodes.go:63\nk8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnAdd\n\t/go/pkg/mod/k8s.io/client-go@v11.0.1-0.20191029005444-8e4128053008+incompatible/tools/cache/controller.go:195\nk8s.io/client-go/tools/cache.(*processorListener).run.func1.1\n\t/go/pkg/mod/k8s.io/client-go@v11.0.1-0.20191029005444-8e4128053008+incompatible/tools/cache/shared_informer.go:554\nk8s.io/apimachinery/pkg/util/wait.ExponentialBackoff\n\t/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20191004074956-c5d2f014d689/pkg/util/wait/wait.go...
RaunakShah commented 3 years ago

Here's the error log:

{"level":"error","time":"2020-12-15T13:01:47.447083828Z","caller":"vsphere/virtualcenter.go:284","msg":"failed to fetch datacenter given dcPath SC5020-06-OpenShift with err: datacenter 'SC5020-06-OpenShift' not found","TraceId":"e0da4bf9-8bf4-4b32-838a-4690d4c77443","stacktrace":"sigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/vsphere.(*VirtualCenter).getDatacenters\n\t/build/pkg/common/cns-lib/vsphere/virtualcenter.go:284\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/vsphere.(*VirtualCenter).GetDatacenters\n\t/build/pkg/common/cns-lib/vsphere/virtualcenter.go:300\nsigs.k8s.io/vsphere-csi-driver/pkg/common/cns-lib/vsphere.asyncGetAllDatacenters\n\t/build/pkg/common/cns-lib/vsphere/datacenter.go:136"}

Are you certain the datacenter SC5020-06-OpenShift is the correct entry?

chapai518 commented 3 years ago

RaunakShah Yes, you are right, we confused concepts datacenter and datastore, also incorrectly indicated the cluster ID.

tmarkunin commented 3 years ago

@RaunakShah Thanks a lot for your help! It works now

RaunakShah commented 3 years ago

@tmarkunin @chapai518 thanks for clarifying! You can visit our documentation for concepts and examples related to vSphere CSI - https://vsphere-csi-driver.sigs.k8s.io/