search

gluster / gluster-csi-driver

DEPRECATED: Gluster Container Storage Interface (CSI) driver
GNU Lesser General Public License v3.0
57 stars 30 forks source link

glusterfs-pv-restore going in in pending state #114

Closed ksandha closed 5 years ago

ksandha commented 6 years ago

Describe the bug After restoring the snapshot the pvc is going in pending state.

[vagrant@kube1 ~]$ kubectl get volumesnapshotclass
NAME                 AGE
glusterfs-csi-snap   2h
[vagrant@kube1 ~]$ 
[vagrant@kube1 ~]$ kubectl get volumesnapshot
No resources found.
[vagrant@kube1 ~]$ 
[vagrant@kube1 ~]$ vi volumesnap.yaml
[vagrant@kube1 ~]$ kubectl create -f volumesnap.yaml 
volumesnapshot.snapshot.storage.k8s.io/glusterfs-csi-ss created
[vagrant@kube1 ~]$ 
[vagrant@kube1 ~]$ kubectl get volumesnapshot
NAME               AGE
glusterfs-csi-ss   20s
[vagrant@kube1 ~]$ 
[vagrant@kube1 ~]$ kubectl get pvc
NAME      STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS    AGE
gcs-pvc   Bound    pvc-b5cdc6df-f480-11e8-802a-525400e8b529   2Gi        RWX            glusterfs-csi   9m20s
[vagrant@kube1 ~]$ 
[vagrant@kube1 ~]$ vi restore.yaml
[vagrant@kube1 ~]$ 
[vagrant@kube1 ~]$ kubectl create -f restore.yaml 
persistentvolumeclaim/glusterfs-pv-restore created
[vagrant@kube1 ~]$ 
[vagrant@kube1 ~]$ kubectl get pvc
NAME                   STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS    AGE
gcs-pvc                Bound     pvc-b5cdc6df-f480-11e8-802a-525400e8b529   2Gi        RWX            glusterfs-csi   9m53s
glusterfs-pv-restore   Pending                                                                        glusterfs-csi   4s
[vagrant@kube1 ~]$ 
[vagrant@kube1 ~]$ kubectl get pvc
NAME                   STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS    AGE
gcs-pvc                Bound     pvc-b5cdc6df-f480-11e8-802a-525400e8b529   2Gi        RWX            glusterfs-csi   84m
glusterfs-pv-restore   Pending                                                                        glusterfs-csi   74m
[vagrant@kube1 ~]$

Actual results glusterfs-restore going in penidng state.

Expected behavior PVC should be in bound state.

Additional context logs from provisioner and CSI attcher

[vagrant@kube1 ~]$ 
[vagrant@kube1 ~]$ kubectl -n gcs logs csi-provisioner-glusterfsplugin-0 csi-provisioner
I1130 07:18:34.251488       1 csi-provisioner.go:82] Version: v0.4.0-rc.1-5-gf2ffed45
I1130 07:18:34.253144       1 csi-provisioner.go:96] Building kube configs for running in cluster...
I1130 07:19:24.295395       1 controller.go:572] Starting provisioner controller org.gluster.glusterfs_csi-provisioner-glusterfsplugin-0_43a1205e-f470-11e8-bcc7-0a580ae94007!
I1130 07:19:24.395931       1 controller.go:621] Started provisioner controller org.gluster.glusterfs_csi-provisioner-glusterfsplugin-0_43a1205e-f470-11e8-bcc7-0a580ae94007!
I1130 09:17:07.857003       1 controller.go:927] provision "default/gcs-pvc" class "glusterfs-csi": started
I1130 09:17:07.910545       1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"gcs-pvc", UID:"b5cdc6df-f480-11e8-802a-525400e8b529", APIVersion:"v1", ResourceVersion:"16769", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/gcs-pvc"
I1130 09:17:09.932893       1 controller.go:551] successfully created PV {GCEPersistentDisk:nil AWSElasticBlockStore:nil HostPath:nil Glusterfs:nil NFS:nil RBD:nil ISCSI:nil Cinder:nil CephFS:nil FC:nil Flocker:nil FlexVolume:nil AzureFile:nil VsphereVolume:nil Quobyte:nil AzureDisk:nil PhotonPersistentDisk:nil PortworxVolume:nil ScaleIO:nil Local:nil StorageOS:nil CSI:&CSIPersistentVolumeSource{Driver:org.gluster.glusterfs,VolumeHandle:pvc-b5cdc6df-f480-11e8-802a-525400e8b529,ReadOnly:false,FSType:ext4,VolumeAttributes:map[string]string{glusterbkpservers: gluster-kube1-0.glusterd2.gcs:gluster-kube2-0.glusterd2.gcs,glusterserver: gluster-kube3-0.glusterd2.gcs,glustervol: pvc-b5cdc6df-f480-11e8-802a-525400e8b529,storage.kubernetes.io/csiProvisionerIdentity: 1543562314290-8081-org.gluster.glusterfs,},ControllerPublishSecretRef:nil,NodeStageSecretRef:nil,NodePublishSecretRef:nil,}}
I1130 09:17:09.933064       1 controller.go:1027] provision "default/gcs-pvc" class "glusterfs-csi": volume "pvc-b5cdc6df-f480-11e8-802a-525400e8b529" provisioned
I1130 09:17:09.933465       1 controller.go:1041] provision "default/gcs-pvc" class "glusterfs-csi": trying to save persistentvolume "pvc-b5cdc6df-f480-11e8-802a-525400e8b529"
I1130 09:17:09.967979       1 controller.go:1048] provision "default/gcs-pvc" class "glusterfs-csi": persistentvolume "pvc-b5cdc6df-f480-11e8-802a-525400e8b529" saved
I1130 09:17:09.968401       1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"gcs-pvc", UID:"b5cdc6df-f480-11e8-802a-525400e8b529", APIVersion:"v1", ResourceVersion:"16769", FieldPath:""}): type: 'Normal' reason: 'ProvisioningSucceeded' Successfully provisioned volume pvc-b5cdc6df-f480-11e8-802a-525400e8b529
I1130 09:17:09.968436       1 controller.go:1089] provision "default/gcs-pvc" class "glusterfs-csi": succeeded
I1130 09:17:09.968553       1 controller.go:927] provision "default/gcs-pvc" class "glusterfs-csi": started
I1130 09:17:09.980924       1 controller.go:936] provision "default/gcs-pvc" class "glusterfs-csi": persistentvolume "pvc-b5cdc6df-f480-11e8-802a-525400e8b529" already exists, skipping
I1130 09:26:56.471669       1 controller.go:927] provision "default/glusterfs-pv-restore" class "glusterfs-csi": started
I1130 09:26:56.516988       1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"glusterfs-pv-restore", UID:"14a84f29-f482-11e8-802a-525400e8b529", APIVersion:"v1", ResourceVersion:"17956", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/glusterfs-pv-restore"
W1130 09:26:56.540666       1 controller.go:686] Retrying syncing claim "default/glusterfs-pv-restore" because failures 0 < threshold 15
E1130 09:26:56.540717       1 controller.go:701] error syncing claim "default/glusterfs-pv-restore": failed to provision volume with StorageClass "glusterfs-csi": error getting snapshot handle for snapshot glusterfs-csi-ss: snapshot glusterfs-csi-ss is not Ready
I1130 09:26:56.540748       1 controller.go:927] provision "default/glusterfs-pv-restore" class "glusterfs-csi": started
I1130 09:26:56.540895       1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"glusterfs-pv-restore", UID:"14a84f29-f482-11e8-802a-525400e8b529", APIVersion:"v1", ResourceVersion:"17956", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "glusterfs-csi": error getting snapshot handle for snapshot glusterfs-csi-ss: snapshot glusterfs-csi-ss is not Ready
I1130 09:26:56.545404       1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"glusterfs-pv-restore", UID:"14a84f29-f482-11e8-802a-525400e8b529", APIVersion:"v1", ResourceVersion:"17957", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/glusterfs-pv-restore"
W1130 09:26:56.569211       1 controller.go:686] Retrying syncing claim "default/glusterfs-pv-restore" because failures 1 < threshold 15
E1130 09:26:56.569278       1 controller.go:701] error syncing claim "default/glusterfs-pv-restore": failed to provision volume with StorageClass "glusterfs-csi": error getting snapshot handle for snapshot glusterfs-csi-ss: snapshot glusterfs-csi-ss is not Ready
I1130 09:26:56.569317       1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"glusterfs-pv-restore", UID:"14a84f29-f482-11e8-802a-525400e8b529", APIVersion:"v1", ResourceVersion:"17957", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "glusterfs-csi": error getting snapshot handle for snapshot glusterfs-csi-ss: snapshot glusterfs-csi-ss is not Ready
I1130 09:27:11.541015       1 controller.go:927] provision "default/glusterfs-pv-restore" class "glusterfs-csi": started
I1130 09:27:11.554804       1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"glusterfs-pv-restore", UID:"14a84f29-f482-11e8-802a-525400e8b529", APIVersion:"v1", ResourceVersion:"17957", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/glusterfs-pv-restore"
W1130 09:27:11.561985       1 controller.go:686] Retrying syncing claim "default/glusterfs-pv-restore" because failures 2 < threshold 15
E1130 09:27:11.562040       1 controller.go:701] error syncing claim "default/glusterfs-pv-restore": failed to provision volume with StorageClass "glusterfs-csi": error getting snapshot handle for snapshot glusterfs-csi-ss: snapshot glusterfs-csi-ss is not Ready
I1130 09:27:11.562106       1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"glusterfs-pv-restore", UID:"14a84f29-f482-11e8-802a-525400e8b529", APIVersion:"v1", ResourceVersion:"17957", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "glusterfs-csi": error getting snapshot handle for snapshot glusterfs-csi-ss: snapshot glusterfs-csi-ss is not Ready
I1130 09:28:11.562373       1 controller.go:927] provision "default/glusterfs-pv-restore" class "glusterfs-csi": started
I1130 09:28:11.569857       1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"glusterfs-pv-restore", UID:"14a84f29-f482-11e8-802a-525400e8b529", APIVersion:"v1", ResourceVersion:"17957", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/glusterfs-pv-restore"
W1130 09:28:11.579321       1 controller.go:686] Retrying syncing claim "default/glusterfs-pv-restore" because failures 3 < threshold 15
E1130 09:28:11.579420       1 controller.go:701] error syncing claim "default/glusterfs-pv-restore": failed to provision volume with StorageClass "glusterfs-csi": error getting snapshot handle for snapshot glusterfs-csi-ss: snapshot glusterfs-csi-ss is not Ready
I1130 09:28:11.580443       1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"glusterfs-pv-restore", UID:"14a84f29-f482-11e8-802a-525400e8b529", APIVersion:"v1", ResourceVersion:"17957", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "glusterfs-csi": error getting snapshot handle for snapshot glusterfs-csi-ss: snapshot glusterfs-csi-ss is not Ready
I1130 09:30:11.579757       1 controller.go:927] provision "default/glusterfs-pv-restore" class "glusterfs-csi": started
I1130 09:30:11.593273       1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"glusterfs-pv-restore", UID:"14a84f29-f482-11e8-802a-525400e8b529", APIVersion:"v1", ResourceVersion:"17957", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/glusterfs-pv-restore"
W1130 09:30:11.598763       1 controller.go:686] Retrying syncing claim "default/glusterfs-pv-restore" because failures 4 < threshold 15
E1130 09:30:11.598811       1 controller.go:701] error syncing claim "default/glusterfs-pv-restore": failed to provision volume with StorageClass "glusterfs-csi": error getting snapshot handle for snapshot glusterfs-csi-ss: snapshot glusterfs-csi-ss is not Ready
I1130 09:30:11.598853       1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"glusterfs-pv-restore", UID:"14a84f29-f482-11e8-802a-525400e8b529", APIVersion:"v1", ResourceVersion:"17957", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "glusterfs-csi": error getting snapshot handle for snapshot glusterfs-csi-ss: snapshot glusterfs-csi-ss is not Ready
I1130 09:34:11.599159       1 controller.go:927] provision "default/glusterfs-pv-restore" class "glusterfs-csi": started
I1130 09:34:11.611202       1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"glusterfs-pv-restore", UID:"14a84f29-f482-11e8-802a-525400e8b529", APIVersion:"v1", ResourceVersion:"17957", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/glusterfs-pv-restore"
W1130 09:34:11.621265       1 controller.go:686] Retrying syncing claim "default/glusterfs-pv-restore" because failures 5 < threshold 15
E1130 09:34:11.621381       1 controller.go:701] error syncing claim "default/glusterfs-pv-restore": failed to provision volume with StorageClass "glusterfs-csi": error getting snapshot handle for snapshot glusterfs-csi-ss: snapshot glusterfs-csi-ss is not Ready
I1130 09:34:11.621447       1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"glusterfs-pv-restore", UID:"14a84f29-f482-11e8-802a-525400e8b529", APIVersion:"v1", ResourceVersion:"17957", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "glusterfs-csi": error getting snapshot handle for snapshot glusterfs-csi-ss: snapshot glusterfs-csi-ss is not Ready
I1130 09:42:11.621668       1 controller.go:927] provision "default/glusterfs-pv-restore" class "glusterfs-csi": started
I1130 09:42:11.635751       1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"glusterfs-pv-restore", UID:"14a84f29-f482-11e8-802a-525400e8b529", APIVersion:"v1", ResourceVersion:"17957", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/glusterfs-pv-restore"
W1130 09:42:11.642282       1 controller.go:686] Retrying syncing claim "default/glusterfs-pv-restore" because failures 6 < threshold 15
E1130 09:42:11.642337       1 controller.go:701] error syncing claim "default/glusterfs-pv-restore": failed to provision volume with StorageClass "glusterfs-csi": error getting snapshot handle for snapshot glusterfs-csi-ss: snapshot glusterfs-csi-ss is not Ready
I1130 09:42:11.642676       1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"glusterfs-pv-restore", UID:"14a84f29-f482-11e8-802a-525400e8b529", APIVersion:"v1", ResourceVersion:"17957", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "glusterfs-csi": error getting snapshot handle for snapshot glusterfs-csi-ss: snapshot glusterfs-csi-ss is not Ready
I1130 09:58:11.642709       1 controller.go:927] provision "default/glusterfs-pv-restore" class "glusterfs-csi": started
I1130 09:58:15.049834       1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"glusterfs-pv-restore", UID:"14a84f29-f482-11e8-802a-525400e8b529", APIVersion:"v1", ResourceVersion:"17957", FieldPath:""}): type: 'Normal' reason: 'Provisioning' External provisioner is provisioning volume for claim "default/glusterfs-pv-restore"
W1130 09:58:15.060774       1 controller.go:686] Retrying syncing claim "default/glusterfs-pv-restore" because failures 7 < threshold 15
E1130 09:58:15.060877       1 controller.go:701] error syncing claim "default/glusterfs-pv-restore": failed to provision volume with StorageClass "glusterfs-csi": error getting snapshot handle for snapshot glusterfs-csi-ss: snapshot glusterfs-csi-ss is not Ready
I1130 09:58:15.060945       1 event.go:221] Event(v1.ObjectReference{Kind:"PersistentVolumeClaim", Namespace:"default", Name:"glusterfs-pv-restore", UID:"14a84f29-f482-11e8-802a-525400e8b529", APIVersion:"v1", ResourceVersion:"17957", FieldPath:""}): type: 'Warning' reason: 'ProvisioningFailed' failed to provision volume with StorageClass "glusterfs-csi": error getting snapshot handle for snapshot glusterfs-csi-ss: snapshot glusterfs-csi-ss is not Ready
[vagrant@kube1 ~]$ 
[vagrant@kube1 ~]$ kubectl -n gcs logs csi-attacher-glusterfsplugin-0 csi-attacher
Error from server: Get https://192.168.121.244:10250/containerLogs/gcs/csi-attacher-glusterfsplugin-0/csi-attacher: net/http: TLS handshake timeout
[vagrant@kube1 ~]$ kubectl -n gcs logs csi-attacher-glusterfsplugin-0 csi-attacher
Error from server: Get https://192.168.121.244:10250/containerLogs/gcs/csi-attacher-glusterfsplugin-0/csi-attacher: net/http: TLS handshake timeout
[vagrant@kube1 ~]$ 
[vagrant@kube1 ~]$
ksandha commented 6 years ago

glusterd2.log

Madhu-1 commented 6 years ago

@ksandha, as discussed over chat, seems like its a user error while creating a snapshot. please feel free to close this issue.

Madhu-1 commented 5 years ago

@ksandha ^^