VCP fails to dynamically provision storage with error NoPermission

[benhwebster]

Is this a BUG REPORT or FEATURE REQUEST?: /kind bug

What happened: VCP is unable to provision dynamic storage, static works fine

What you expected to happen: VCP to successfully provision storage dynamically

How to reproduce it (as minimally and precisely as possible): Set up VCP according to documentation here, using the vSphere permissions here. Create storage class, create a claim using the storage class, claim fails to provision storage with error NoPermission

Anything else we need to know?: relevent logs(?):

I0924 13:13:12.911381       1 vsphere_util.go:123] error "NoPermission" getting accessible datastores for node &{NodeName:infranode1 vm:0xc42720f2b0 VMUUID:}
E0924 13:13:12.911391       1 vsphere.go:1077] Failed to get shared datastore: NoPermission
I0924 13:13:12.911400       1 vsphere.go:1129] The canonical volume path for the newly created vSphere volume is ""
I0924 13:13:12.911413       1 pv_controller.go:1433] failed to provision volume for claim "test-project/sdfawefw" with StorageClass "vsphere-nprd": NoPermission
I0924 13:13:12.911445       1 event.go:218] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"test-project", Name:"sdfawefw", UID:"26a10844-bffb-11e8-beda-005056964b02", APIVersion:"v1", ResourceVersion:"1579982", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' Failed to provision volume with StorageClass "vsphere-nprd": NoPermission

Static provisioning works, (create disk, create pv, create pvc) but something about the dynamic provisioning portion is missing privileges and I don't know what or how to find out what it is.

Possibly related to https://github.com/kubernetes/kubernetes/issues/67206?

Environment:

• Kubernetes version (use kubectl version): Openshift 3.10 Client Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.0+b81c8f8", GitCommit:"b81c8f8", GitTreeState:"clean", BuildDate:"2018-09-07T19:14:28Z", GoVersion:"go1.9.4", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"10+", GitVersion:"v1.10.0+b81c8f8", GitCommit:"b81c8f8", GitTreeState:"clean", BuildDate:"2018-09-07T19:14:28Z", GoVersion:"go1.9.4", Compiler:"gc", Platform:"linux/amd64"}
• Cloud provider or hardware configuration: vSphere ESXi 6.5 (VM version 13)
• OS (e.g. from /etc/os-release): Red Hat 7.5
• Kernel (e.g. uname -a): 3.10.0-862.14.4.el7.x86_64 #1 SMP Fri Sep 21 09:07:21 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
• Install tools: Openshift-ansible playbook
• Others: vsphere.conf vsphere-nprd.yaml

[goebi]

Any news here? We have also the issue with dynamically provision storage not working.

[benhwebster]

No sadly, I've pretty much given up on the dynamic portion, our VMware admins think the account is set up correctly according to the permissions required for dynamic provisioning without storage policy based placement, and they don't want to give the account the permissions required for that so I'm stuck.

[divyenpatel]

@benhwebster

Do you have System.View privilege on the user. This is required to call RetrieveProperties

This is failing at https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/vsphere/vclib/virtualmachine.go#L214

[benhwebster]

Finally found the missing permission! we were missing read-only permission on one of the ESXi clusters running the hosts running the infrastructure nodes (or one of the children of the cluster, we let it propagate, you could probably just add ESXi cluster as one of entities for the read-only role in those tables.) Thank you Divyen for pointing me in the right direction!