Dynamic provisioning not creating PV from default storage class

[sudhakar-mnsr]

Is this a BUG REPORT or FEATURE REQUEST?:

Uncomment only one, leave it on its own line:

/kind bug /kind feature

What happened: Iam unable to bind storage class and provision volume dynamically for a claim.

What you expected to happen: PVC should get PV from default storage class. How to reproduce it (as minimally and precisely as possible): Steps Iam following:

Iam using non-admin user and creating VM’s in vsphere environment. I have kubernetes cluster setup (with one node). I have NOT used kubernetes anywhere for deploying the cluster.

I have enabled DefaultStorageClass in the admission control options

Added cloud-provider=vsphere and cloud-config file to apiserver, control-manager and kubelet

Created storage class kind: StorageClass apiVersion: storage.k8s.io/v1 metadata: name: fast provisioner: kubernetes.io/vsphere-volume parameters: diskformat: zeroedthick fstype: ext3

Output: Name: fast IsDefaultClass: Yes Annotations: storageclass.kubernetes.io/is-default-class=true Provisioner: kubernetes.io/vsphere-volume Parameters: diskformat=zeroedthick,fstype=ext3 Events:

Made it default kubectl patch storageclass fast -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

Created PVC as below. It is not getting PV. And the status is pending kind: PersistentVolumeClaim apiVersion: v1 metadata: name: pvcsc001 spec: accessModes:

• ReadWriteOnce resources: requests: storage: 2Gi

Output: kubectl describe pvc pvcsc001

Name: pvcsc001 Namespace: default StorageClass: fast Status: Pending Volume: Labels: Annotations: Capacity: Access Modes: Events:

Am I missing anything please suggest. Help me in debugging the issue.

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version): 1.7.2
• Cloud provider or hardware configuration**:
• OS (e.g. from /etc/os-release): SLES12 SP2
• Kernel (e.g. uname -a):
• Install tools:
• Others:

[sudhakar-mnsr]

Checked the snapshots you have provided. You have Administrator role set on the default datastore.

For non-administrator user, recommendation is to create roles as mentioned at - https://kubernetes.io/docs/getting-started-guides/vsphere/

Video walkthrough on How to create role is available at https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.security.doc/GUID-18071E9A-EED1-4968-8D51-E0B4F526FDA3.html

Can you share the controller logs? Would be good to track this on github. Please create an issue and attach logs.

$ kubectl get pods --namespace=kube-system | grep controller kube-controller-manager-kubernetes-master 1/1 Running 3 4h

$ kubectl logs kube-controller-manager-kubernetes-master --namespace=kube-system

if due to tightened security, logs cannot be accessible using kubectl, you can ssh into kubernetes master and collect logs from the controller container. Logs are generally available at /var/lib/docker/containers//

I can setup a quick WebEx meeting to validate your setup. Can you provide the timeslots?

Thanks for your help and good pointers.

I have run the commands and got "No Resources found" linux-3xcy:~ # kubectl --kubeconfig=/var/lib/kubelet/kubeconfig get pods --namespace=kube-system 2017-10-16 02:32:29.853590 I | proto: duplicate proto type registered: google.protobuf.Any 2017-10-16 02:32:29.853660 I | proto: duplicate proto type registered: google.protobuf.Duration 2017-10-16 02:32:29.853675 I | proto: duplicate proto type registered: google.protobuf.Timestamp No resources found.

Further debugging I have removed the Flags --cloud-provider=vsphere, --cloud-config=/tmp/vsphere.conf (in apiserver controller-manager kubelet) everything is fine and could see containers for the above command. Next iteration I just added one flag--cloud-provider=vsphere and I could see no resources found message.

One more Observation: Even in successful case (without flags --cloud-provider=vsphere, --cloud-config=/tmp/vsphere.conf) the below command gave error linux-3xcy:~ # kubectl logs kube-controller-manager-kubernetes-master --namespace=kube-system Error from server (NotFound): pods "kube-controller-manager-kubernetes-master" not found

However I would like to know that till now I have not created any roles. Iam under the impression that Administrator role implies all the roles mentioned in document. If i have to explicitly create these roles Can i create them with non-admin user?

[divyenpatel]

If i have to explicitly create these roles Can i create them with non-admin user?

You should create new roles using administrator account.

I am under the impression that Administrator role implies all the roles mentioned in document.

Make sure permission is assigned to the user with the administrator role at all levels, vCenter, Datacenter, Cluster, Hosts, Datastore.

Please upload controller-manager logs here on the github to help us debug this issue further. Login in to the master node. Find container id of the controller-manager node using docker ps | grep controller-manager, and go to /var/lib/docker/containers/<controller-manager-container-id>/.

sample instructions on how to get log files.

root@kubernetes-master [ ~ ]# docker ps | grep controller-manager
0dae71ecc856        2215c1cdc3b5                                         "/hyperkube controlle"   15 hours ago        Up 15 hours                             k8s_kube-controller-manager_kube-controller-manager-kubernetes-master_kube-system_8f44e7e7a848960717b6031d18f6ffa9_2
73530605e852        gcr.io/google_containers/pause-amd64:3.0             "/pause"                 20 hours ago        Up 20 hours                             k8s_POD_kube-controller-manager-kubernetes-master_kube-system_8f44e7e7a848960717b6031d18f6ffa9_0

root@kubernetes-master [ ~ ]# cd /var/lib/docker/containers/0dae71ecc856950b3dd00f447164dc05a345b2ea1a60180795ec8ff656d52d17/

root@kubernetes-master [ /var/lib/docker/containers/0dae71ecc856950b3dd00f447164dc05a345b2ea1a60180795ec8ff656d52d17 ]# ls -la
total 1158108
drwx------  2 root root       4096 Oct 16 12:52 .
drwx------ 18 root root       4096 Oct 16 12:53 ..
-rw-r-----  1 root root 1185876080 Oct 17 03:58 0dae71ecc856950b3dd00f447164dc05a345b2ea1a60180795ec8ff656d52d17-json.log
-rw-r--r--  1 root root       5805 Oct 16 12:52 config.v2.json
-rw-r--r--  1 root root       1577 Oct 16 12:52 hostconfig.jsonx

cc: @BaluDontu @tusharnt @pshahzeb

[sudhakar-mnsr]

We have process to get these roles for my user. I will update when done.

[divyenpatel]

@sudhakar-mnsr We have automated the process for creating roles and assigning permission. you can try this out - https://github.com/vmware/kubernetes/tree/enable-vcp-uxi

but before that, if you can get us the controller logs, we can quickly identify where the problem is in your setup.

[sudhakar-mnsr]

As mentioned above if I use cloud-provider option i am not seeing any containers in /var/lib/docker also the commands give "No Resources found". Please let me know if you want any other logs/information.

I dont have admin privilleges so I can suggest my admin to use it. Thanks

[divyenpatel]

Can you provide kubelet logs with keeping cloud-provider options?

[sudhakar-mnsr]

Please let me know how to get them. I did not see anything like kubelet.log in /var/log.

[sudhakar-mnsr]

journalctl.txt

[divyenpatel]

The issue is in the cloud provider initialization.

Please keep cloud provider options, and add vm-name to the vsphere.conf file. Make this change in all nodes and restart kubelet and controller services.

Global]
       .
       .
        vm-name = "node-vm-name-in-the-vc"
[Disk]
        scsicontrollertype = pvscsi

if you can also increase the log level (--v=9), we can identify the line where the error is occurring in the cloud provider initialization.

[sudhakar-mnsr]

I have changed the conf file to have vm-name and could see containers new journalctl log is attached journalctl2.txt.txt

However few observations/clarifications As you have seen previous snapshots user sm185078 has Administrator role so does that mean user sm185078 is an admin user? The controller-manager is not running as container it is just running as daemon process

linux-3xcy # docker ps | grep -i "controller-manager" linux-3xcy # ps -aef | grep -i controller-manager kube 13672 1 0 14:54 ? 00:00:18 /usr/bin/kube-controller-manager --logtostderr=true --v=1 --master=https://10.25.151.56:8081 --allocate-node-cidrs=true --kubeconfig=/var/lib/kubelet/kubeconfig --cloud-provider=vsphere --cloud-config=/tmp/vsphere.conf --leader-elect=true --cluster-name=kubernetes --cluster-cidr=172.24.0.0/16 --service-account-private-key-file=/etc/pki/node.key --root-ca-file=/etc/pki/trust/anchors/root_ca.crt --v=2

My admins are under the discussion that user sm185078 is admin (i have put my points) so i have not set the roles with privileges. just gave a try to create storage class, set it default, created PVC. still the same result (pending).

[divyenpatel]

Can you provide controller logs using journalctl -u kube-controller-manager.service ?

[sudhakar-mnsr]

journalctl3.txt.txt

[divyenpatel]

Now we have the right information to help you.

You are trying to provision the volume with specifying storage policy name in the storage class, but seems profile is not available in your vCenter.

vsphere_volume_util.go:145] Error creating vsphere volume: no pbm profile found with name: "gold"

please use following yamls.

$ cat thin_sc.yaml 
kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
metadata:
  name: thin
provisioner: kubernetes.io/vsphere-volume
parameters:
    diskformat: thin

$ cat thin_pvc.yaml 
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: thinclaim
  annotations:
    volume.beta.kubernetes.io/storage-class: thin
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 2Gi

$ cat thin_pod.yaml 
apiVersion: v1
kind: Pod
metadata:
    name: thinclaimpod
spec:
    containers:
    - name: test-container
      image: gcr.io/google_containers/busybox:1.24
      command: ["/bin/sh", "-c", "echo 'hello' > /mnt/volume1/index.html  && chmod o+rX /mnt /mnt/volume1/index.html && while true ; do sleep 2 ; done"]
      volumeMounts:
      - name: test-volume
        mountPath: /mnt/volume1
    restartPolicy: Never
    volumes:
    - name: test-volume
      persistentVolumeClaim:
        claimName: thinclaim

[divyenpatel]

@sudhakar-mnsr as discussed in the web ex meeting, since you are able to provision the PV dynamically using storage class, I am closing this issue.

[sanjeevshar]

Please pardon my ignorance as I am new to Pivotal PKS and roles and permissions in Vsphere. I am having trouble creating static and dunamic PVs. As mentioned in documentation https://vmware.github.io/vsphere-storage-for-kubernetes/documentation/vcp-roles.html, I created roles manage-k8s-node-vms and manage-k8s-volumes. My questions are:

1. Do I create two new users and assign one role to each?
2. Pivotal Container Service in Kubernetes Cloud Provider Configuration requires vCenter Master Credentials and vCenter Worker Credentials. Which role user goes in which setting ?