search

kubernetes / cloud-provider-vsphere

Kubernetes Cloud Provider for vSphere https://cloud-provider-vsphere.sigs.k8s.io
Apache License 2.0
242 stars 177 forks source link

Dynamically provisioned Persistence Volume is not getting bound with its PV Claim. #135

Closed divyenpatel closed 5 years ago

divyenpatel commented 5 years ago

Is this a BUG REPORT or FEATURE REQUEST?:

/kind bug

What happened: Dynamically provisioned Persistence Volume is not getting bound with its PV Claim.

Storage Class

# cat sc.yaml 
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: my-vsphere-fcd-class
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: io.k8s.cloud-provider-vsphere.vsphere
parameters:
  parent_type: "Datastore"
  parent_name: "sharedVmfs-0"

Persistence Volume Claim

# cat pvc.yaml 
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-vsphere-csi-pvc
spec:
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi
  storageClassName: my-vsphere-fcd-class
# kubectl create -f sc.yaml 
storageclass.storage.k8s.io/my-vsphere-fcd-class created

# kubectl create -f pvc.yaml 
persistentvolumeclaim/my-vsphere-csi-pvc created

PV Claim and PV remains in the Pending State and does not get bound.

# kubectl get pvc
NAME                 STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS           AGE
my-vsphere-csi-pvc   Pending                                      my-vsphere-fcd-class   21s

# kubectl describe pvc my-vsphere-csi-pvc
Name:          my-vsphere-csi-pvc
Namespace:     default
StorageClass:  my-vsphere-fcd-class
Status:        Pending
Volume:        
Labels:        <none>
Annotations:   volume.beta.kubernetes.io/storage-provisioner: io.k8s.cloud-provider-vsphere.vsphere
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Filesystem
Events:
  Type       Reason                 Age                From                                                                                                 Message
  ----       ------                 ----               ----                                                                                                 -------
  Normal     Provisioning           55s                io.k8s.cloud-provider-vsphere.vsphere_vsphere-csi-controller-0_a0ab202d-243b-11e9-90a7-0a580af40102  External provisioner is provisioning volume for claim "default/my-vsphere-csi-pvc"
  Normal     ProvisioningSucceeded  14s                io.k8s.cloud-provider-vsphere.vsphere_vsphere-csi-controller-0_a0ab202d-243b-11e9-90a7-0a580af40102  Successfully provisioned volume pvc-aa2fd750-24b9-11e9-9c0e-005056a43de5
  Normal     ExternalProvisioning   10s (x5 over 55s)  persistentvolume-controller                                                                          waiting for a volume to be created, either by external provisioner "io.k8s.cloud-provider-vsphere.vsphere" or manually created by system administrator
Mounted By:  my-csi-app
# kubectl describe pv pvc-aa2fd750-24b9-11e9-9c0e-005056a43de5
Name:            pvc-aa2fd750-24b9-11e9-9c0e-005056a43de5
Labels:          <none>
Annotations:     pv.kubernetes.io/provisioned-by: io.k8s.cloud-provider-vsphere.vsphere
Finalizers:      []
StorageClass:    my-vsphere-fcd-class
Status:          Pending
Claim:           default/my-vsphere-csi-pvc
Reclaim Policy:  Delete
Access Modes:    RWO
VolumeMode:      Filesystem
Capacity:        5Gi
Node Affinity:   <none>
Message:         
Source:
    Type:              CSI (a Container Storage Interface (CSI) volume source)
    Driver:            io.k8s.cloud-provider-vsphere.vsphere
    VolumeHandle:      0cb048c8-0f2a-4b8a-bbc4-8f1fec1b1fe6
    ReadOnly:          false
    VolumeAttributes:      datacenter=vcqaDC
                           name=pvc-aa2fd750-24b9-11e9-9c0e-005056a43de5
                           parent_name=sharedVmfs-0
                           parent_type=Datastore
                           storage.kubernetes.io/csiProvisionerIdentity=1548817403656-8081-io.k8s.cloud-provider-vsphere.vsphere
                           type=First Class Disk
                           vcenter=10.161.155.44
Events:                <none>

Logs

csi-provisioner logs

I0130 18:05:52.034637       1 controller.go:605] create volume rep: {CapacityBytes:5368709120 VolumeId:0cb048c8-0f2a-4b8a-bbc4-8f1fec1b1fe6 VolumeContext:map[parent_name:sharedVmfs-0 type:First Class Disk vcenter:10.161.155.44 datacenter:vcqaDC name:pvc-aa2fd750-24b9-11e9-9c0e-005056a43de5 parent_type:Datastore] ContentSource:<nil> AccessibleTopology:[] XXX_NoUnkeyedLiteral:{} XXX_unrecognized:[] XXX_sizecache:0}
I0130 18:05:52.034715       1 controller.go:666] successfully created PV {GCEPersistentDisk:nil AWSElasticBlockStore:nil HostPath:nil Glusterfs:nil NFS:nil RBD:nil ISCSI:nil Cinder:nil CephFS:nil FC:nil Flocker:nil FlexVolume:nil AzureFile:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil PortworxVolume:nil ScaleIO:nil Local:nil StorageOS:nil CSI:&CSIPersistentVolumeSource{Driver:io.k8s.cloud-provider-vsphere.vsphere,VolumeHandle:0cb048c8-0f2a-4b8a-bbc4-8f1fec1b1fe6,ReadOnly:false,FSType:ext4,VolumeAttributes:map[string]string{datacenter: vcqaDC,name: pvc-aa2fd750-24b9-11e9-9c0e-005056a43de5,parent_name: sharedVmfs-0,parent_type: Datastore,storage.kubernetes.io/csiProvisionerIdentity: 1548817403656-8081-io.k8s.cloud-provider-vsphere.vsphere,type: First Class Disk,vcenter: 10.161.155.44,},ControllerPublishSecretRef:nil,NodeStageSecretRef:nil,NodePublishSecretRef:nil,}}
I0130 18:05:52.034819       1 controller.go:1026] provision "default/my-vsphere-csi-pvc" class "my-vsphere-fcd-class": volume "pvc-aa2fd750-24b9-11e9-9c0e-005056a43de5" provisioned
I0130 18:05:52.034836       1 controller.go:1040] provision "default/my-vsphere-csi-pvc" class "my-vsphere-fcd-class": trying to save persistentvolume "pvc-aa2fd750-24b9-11e9-9c0e-005056a43de5"
I0130 18:06:22.040682       1 controller.go:1052] provision "default/my-vsphere-csi-pvc" class "my-vsphere-fcd-class": failed to save persistentvolume "pvc-aa2fd750-24b9-11e9-9c0e-005056a43de5": Timeout: request did not complete within requested timeout 30s
I0130 18:06:32.041055       1 controller.go:1040] provision "default/my-vsphere-csi-pvc" class "my-vsphere-fcd-class": trying to save persistentvolume "pvc-aa2fd750-24b9-11e9-9c0e-005056a43de5"
I0130 18:06:32.046834       1 controller.go:1044] provision "default/my-vsphere-csi-pvc" class "my-vsphere-fcd-class": persistentvolume "pvc-aa2fd750-24b9-11e9-9c0e-005056a43de5" already exists, reusing
I0130 18:06:32.046992       1 controller.go:1088] provision "default/my-vsphere-csi-pvc" class "my-vsphere-fcd-class": succeeded
I0130 18:06:32.047058       1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"my-vsphere-csi-pvc", UID:"aa2fd750-24b9-11e9-9c0e-005056a43de5", APIVersion:"v1", ResourceVersion:"115696", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-aa2fd750-24b9-11e9-9c0e-005056a43de5

vsphere-csi-controller logs

ERROR: logging before flag.Parse: W0130 18:05:51.189849       1 connection.go:77] Creating new client session since the existing session is not valid or not authenticated
ERROR: logging before flag.Parse: E0130 18:05:51.580279       1 datacenter.go:520] GetFirstClassDiskByName failed. Err: No vSphere disk ID/Name found

kube-controller-manager logs

I0130 18:12:06.149093       1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"my-vsphere-csi-pvc", UID:"aa2fd750-24b9-11e9-9c0e-005056a43de5", APIVersion:"v1", ResourceVersion:"115696", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "io.k8s.cloud-provider-vsphere.vsphere" or manually created by system administrator
I0130 18:12:21.149188       1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"my-vsphere-csi-pvc", UID:"aa2fd750-24b9-11e9-9c0e-005056a43de5", APIVersion:"v1", ResourceVersion:"115696", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "io.k8s.cloud-provider-vsphere.vsphere" or manually created by system administrator
I0130 18:12:36.149247       1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"my-vsphere-csi-pvc", UID:"aa2fd750-24b9-11e9-9c0e-005056a43de5", APIVersion:"v1", ResourceVersion:"115696", FieldPath:""}): type: 'Normal' reason: 'ExternalProvisioning' waiting for a volume to be created, either by external provisioner "io.k8s.cloud-provider-vsphere.vsphere" or manually created by system administrator

What you expected to happen:

PV Claim should get bind with dynamically created PV.

Environment:

kube-controller-manager: k8s.gcr.io/kube-controller-manager:v1.13.2

vsphere-csi-controller: quay.io/k8scsi/csi-provisioner:v1.0.1

vsphere-cloud-controller-manager: gcr.io/cloud-provider-vsphere/vsphere-cloud-controller-manager:latest

vsphere-csi-node: quay.io/k8scsi/csi-node-driver-registrar:v1.0.1

davidvonthenen commented 5 years ago

Taking a look at this...

/assign dvonthenen

davidvonthenen commented 5 years ago

Tested using the latest PR https://github.com/kubernetes/cloud-provider-vsphere/pull/136 and wasn't able to find any issues. It could be the same permissions related issue that was updated in the YAML within that PR.

[vonthd@k8smaster csi]$ kubectl create -f example-vsphere-sc.yaml
storageclass.storage.k8s.io/vsphere-fcd created
[vonthd@k8smaster csi]$ kubectl create -f example-vsphere-pvc.yaml
persistentvolumeclaim/vsphere-csi-pvc created
[vonthd@k8smaster csi]$ kubectl get pvc
NAME              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
vsphere-csi-pvc   Bound    pvc-a16daa6b-2afe-11e9-84f7-005056a741a8   5Gi        RWO            vsphere-fcd    7s
[vonthd@k8smaster csi]$ kubectl describe pvc vsphere-csi-pvc
Name:          vsphere-csi-pvc
Namespace:     default
StorageClass:  vsphere-fcd
Status:        Bound
Volume:        pvc-a16daa6b-2afe-11e9-84f7-005056a741a8
Labels:        <none>
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: io.k8s.cloud-provider-vsphere.vsphere
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      5Gi
Access Modes:  RWO
VolumeMode:    Filesystem
Events:
  Type       Reason                 Age                From                                                                                                 Message
  ----       ------                 ----               ----                                                                                                 -------
  Normal     ExternalProvisioning   11s (x2 over 12s)  persistentvolume-controller                                                                          waiting for a volume to be created, either by external provisioner "io.k8s.cloud-provider-vsphere.vsphere" or manually created by system administrator
  Normal     Provisioning           10s                io.k8s.cloud-provider-vsphere.vsphere_vsphere-csi-controller-0_f42e229c-2afd-11e9-bfb0-0e4231c9c60a  External provisioner is provisioning volume for claim "default/vsphere-csi-pvc"
  Normal     ProvisioningSucceeded  8s                 io.k8s.cloud-provider-vsphere.vsphere_vsphere-csi-controller-0_f42e229c-2afd-11e9-bfb0-0e4231c9c60a  Successfully provisioned volume pvc-a16daa6b-2afe-11e9-84f7-005056a741a8
Mounted By:  <none>
[vonthd@k8smaster csi]$
davidvonthenen commented 5 years ago

PR https://github.com/kubernetes/cloud-provider-vsphere/pull/136 was merged a few days ago. Please give it a try and see if it fixes your issue. Note the instructions below. 1) reinstall the CSI driver by grabbing/updating the YAML (fixed permissions) in its deployment 2) the YAML deploys v0.1.1 of the driver… so you will need to change the image to latest within the YAML in 2 places until a new tag becomes available 3) fairly detailed documentation was created for multi-vc/dc and zones support: https://github.com/kubernetes/cloud-provider-vsphere/blob/master/docs/deploying_ccm_and_csi_with_multi_dc_vc_aka_zones.md

divyenpatel commented 5 years ago

Issue is resolved after removing Initializers from enable-admission-plugins flag on API server and removing initializerconfiguration.admissionregistration.k8s.io/pvlabel.kubernetes.io.

I was following instruction mentioned below

I can now see PV and PVC are in the bound state

# kubectl get pvc
NAME                 STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS           AGE
my-vsphere-csi-pvc   Bound    pvc-48a3a1e3-3177-11e9-a630-00505690e9f4   5Gi        RWO            my-vsphere-fcd-class   14m

# kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                        STORAGECLASS           REASON   AGE
pvc-48a3a1e3-3177-11e9-a630-00505690e9f4   5Gi        RWO            Delete           Bound    default/my-vsphere-csi-pvc   my-vsphere-fcd-class            13m
divyenpatel commented 5 years ago

Also made sure pod is up and running using the volume provisioned using vSpehre CSI driver.

# kubectl get pods
NAME         READY   STATUS    RESTARTS   AGE
my-csi-app   1/1     Running   0          13m