Open ksandha opened 5 years ago
1) Created 50 PVC parallel requests. 2) While the pvc are getting processed , rebooted second node from the 3 node GCS cluster.
[vagrant@kube1 ~]$ kubectl get pods -n gcs NAME READY STATUS RESTARTS AGE csi-attacher-glusterfsplugin-0 2/2 Running 0 12m csi-nodeplugin-glusterfsplugin-69jvq 2/2 Running 0 12m csi-nodeplugin-glusterfsplugin-bf5kx 2/2 Running 0 12m csi-nodeplugin-glusterfsplugin-n7wzw 0/2 Error 0 12m csi-provisioner-glusterfsplugin-0 3/3 Running 0 12m etcd-9dnxxnvhq2 1/1 Running 0 18m etcd-9wz7txxx59 1/1 Running 0 16m etcd-b92cpssvmz 0/1 Error 0 17m etcd-operator-7cb5bd459b-mczrk 1/1 Running 0 18m gluster-kube1-0 1/1 Running 1 16m gluster-kube2-0 0/1 Error 1 16m gluster-kube3-0 1/1 Running 1 16m
3) Observing the pvc stuck at 38 for the time being.
[vagrant@kube1 ~]$ kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-d92e016d-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc0 glusterfs-csi 116s pvc-d959ddbe-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc1 glusterfs-csi 115s pvc-d9860619-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc2 glusterfs-csi 2m19s pvc-d9b5e137-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc3 glusterfs-csi 2m19s pvc-da1ac34c-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc5 glusterfs-csi 2m18s pvc-da510fc2-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc6 glusterfs-csi 2m18s pvc-db002280-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc9 glusterfs-csi 2m17s pvc-db342a77-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc10 glusterfs-csi 2m15s pvc-db66d2fe-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc11 glusterfs-csi 119s pvc-db9bcff5-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc12 glusterfs-csi 2m16s pvc-dbce239d-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc13 glusterfs-csi 114s pvc-dc04ef7a-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc14 glusterfs-csi 2m15s pvc-dc39d4bc-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc15 glusterfs-csi 2m14s pvc-dca4b450-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc17 glusterfs-csi 2m13s pvc-dcddb56f-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc18 glusterfs-csi 2m12s pvc-dd136269-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc19 glusterfs-csi 2m12s pvc-dd459ffd-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc20 glusterfs-csi 118s pvc-ddaed68a-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc22 glusterfs-csi 116s pvc-dde6b8bb-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc23 glusterfs-csi 2m11s pvc-de4d84fc-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc25 glusterfs-csi 2m9s pvc-de8758b5-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc26 glusterfs-csi 119s pvc-dec2600c-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc27 glusterfs-csi 2m9s pvc-def3a474-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc28 glusterfs-csi 2m9s pvc-df2e25d9-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc29 glusterfs-csi 2m8s pvc-df64fa77-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc30 glusterfs-csi 2m6s pvc-df99124a-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc31 glusterfs-csi 2m6s pvc-dfce5481-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc32 glusterfs-csi 2m5s pvc-e0066f0b-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc33 glusterfs-csi 2m4s pvc-e0431efb-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc34 glusterfs-csi 2m3s pvc-e0925da6-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc35 glusterfs-csi 2m3s pvc-e0c314f3-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc36 glusterfs-csi 2m2s pvc-e0f896db-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc37 glusterfs-csi 2m1s pvc-e12be8b5-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc38 glusterfs-csi 119s
4) Eventually the gd2 pods and csi nodeplugin came up and running, but the etcd remain in the same state.
[vagrant@kube1 ~]$ kubectl get pods -n gcs NAME READY STATUS RESTARTS AGE csi-attacher-glusterfsplugin-0 2/2 Running 0 15m csi-nodeplugin-glusterfsplugin-69jvq 2/2 Running 0 15m csi-nodeplugin-glusterfsplugin-bf5kx 2/2 Running 0 15m csi-nodeplugin-glusterfsplugin-n7wzw 2/2 Running 2 15m csi-provisioner-glusterfsplugin-0 3/3 Running 0 15m etcd-9dnxxnvhq2 1/1 Running 0 21m etcd-9wz7txxx59 1/1 Running 0 19m etcd-b92cpssvmz 0/1 Error 0 20m etcd-operator-7cb5bd459b-mczrk 1/1 Running 0 21m gluster-kube1-0 1/1 Running 1 19m gluster-kube2-0 1/1 Running 3 19m gluster-kube3-0 1/1 Running 1 19m
5) Wait for 10 mins, the etcd pod gets killed and a new pod is spun
[vagrant@kube1 ~]$ kubectl get pods -n gcs NAME READY STATUS RESTARTS AGE csi-attacher-glusterfsplugin-0 2/2 Running 0 22m csi-nodeplugin-glusterfsplugin-69jvq 2/2 Running 0 22m csi-nodeplugin-glusterfsplugin-bf5kx 2/2 Running 0 22m csi-nodeplugin-glusterfsplugin-n7wzw 2/2 Running 2 22m csi-provisioner-glusterfsplugin-0 3/3 Running 0 22m etcd-9dnxxnvhq2 1/1 Running 0 28m etcd-9wz7txxx59 1/1 Running 0 26m etcd-dv6q2n7c27 0/1 Init:0/1 0 26s etcd-operator-7cb5bd459b-mczrk 1/1 Running 0 28m gluster-kube1-0 1/1 Running 1 26m gluster-kube2-0 1/1 Running 3 26m gluster-kube3-0 1/1 Running 1 26m [vagrant@kube1 ~]$
6) The cluster state is in healthy state.
[vagrant@kube1 ~]$ kubectl get pods -n gcs NAME READY STATUS RESTARTS AGE csi-attacher-glusterfsplugin-0 2/2 Running 0 22m csi-nodeplugin-glusterfsplugin-69jvq 2/2 Running 0 22m csi-nodeplugin-glusterfsplugin-bf5kx 2/2 Running 0 22m csi-nodeplugin-glusterfsplugin-n7wzw 2/2 Running 2 22m csi-provisioner-glusterfsplugin-0 3/3 Running 0 22m etcd-9dnxxnvhq2 1/1 Running 0 28m etcd-9wz7txxx59 1/1 Running 0 27m etcd-dv6q2n7c27 1/1 Running 0 54s etcd-operator-7cb5bd459b-mczrk 1/1 Running 0 29m gluster-kube1-0 1/1 Running 1 27m gluster-kube2-0 1/1 Running 3 27m gluster-kube3-0 1/1 Running 1 27m [vagrant@kube1 ~]$ [vagrant@kube1 ~]$ [vagrant@kube1 ~]$ [vagrant@kube1 ~]$ [vagrant@kube1 ~]$ [vagrant@kube1 ~]$ kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE pvc-d92e016d-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc0 glusterfs-csi 11m pvc-d959ddbe-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc1 glusterfs-csi 11m pvc-d9860619-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc2 glusterfs-csi 11m pvc-d9b5e137-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc3 glusterfs-csi 11m pvc-d9ec5839-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc4 glusterfs-csi 80s pvc-da1ac34c-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc5 glusterfs-csi 11m pvc-da510fc2-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc6 glusterfs-csi 11m pvc-da95c9c6-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc7 glusterfs-csi 76s pvc-db002280-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc9 glusterfs-csi 11m pvc-db342a77-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc10 glusterfs-csi 11m pvc-db66d2fe-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc11 glusterfs-csi 11m pvc-db9bcff5-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc12 glusterfs-csi 11m pvc-dbce239d-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc13 glusterfs-csi 11m pvc-dc04ef7a-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc14 glusterfs-csi 11m pvc-dc39d4bc-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc15 glusterfs-csi 11m pvc-dc6dc420-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc16 glusterfs-csi 64s pvc-dca4b450-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc17 glusterfs-csi 11m pvc-dcddb56f-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc18 glusterfs-csi 11m pvc-dd136269-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc19 glusterfs-csi 11m pvc-dd459ffd-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc20 glusterfs-csi 11m pvc-dd7b42c5-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc21 glusterfs-csi 77s pvc-ddaed68a-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc22 glusterfs-csi 11m pvc-dde6b8bb-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc23 glusterfs-csi 11m pvc-de1a6f0f-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc24 glusterfs-csi 80s pvc-de4d84fc-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc25 glusterfs-csi 11m pvc-de8758b5-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc26 glusterfs-csi 11m pvc-dec2600c-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc27 glusterfs-csi 11m pvc-def3a474-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc28 glusterfs-csi 11m pvc-df2e25d9-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc29 glusterfs-csi 11m pvc-df64fa77-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc30 glusterfs-csi 11m pvc-df99124a-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc31 glusterfs-csi 11m pvc-dfce5481-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc32 glusterfs-csi 11m pvc-e0066f0b-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc33 glusterfs-csi 11m pvc-e0431efb-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc34 glusterfs-csi 11m pvc-e0925da6-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc35 glusterfs-csi 11m pvc-e0c314f3-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc36 glusterfs-csi 11m pvc-e0f896db-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc37 glusterfs-csi 11m pvc-e12be8b5-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc38 glusterfs-csi 11m pvc-e1610f73-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc39 glusterfs-csi 85s pvc-e1cecca8-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc41 glusterfs-csi 85s pvc-e2072dd3-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc42 glusterfs-csi 84s pvc-e23c35fa-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc43 glusterfs-csi 82s pvc-e2764934-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc44 glusterfs-csi 79s pvc-e2b1a377-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc45 glusterfs-csi 82s pvc-e2eefa00-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc46 glusterfs-csi 81s pvc-e326ff76-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc47 glusterfs-csi 80s pvc-e35b14fa-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc48 glusterfs-csi 78s pvc-e39289e9-0e6a-11e9-a7a5-525400ee92d8 2Gi RWX Delete Bound default/gcs-pvc49 glusterfs-csi 78s
etcd-9dnxxnvhq2.log etcd-9wz7txxx59.log etcd-dv6q2n7c27.log
rhsqa-virt05.lab.eng.blr.redhat.com:/root/isse95_gd2.log glusterd2 logs placed at above location
1) Created 50 PVC parallel requests. 2) While the pvc are getting processed , rebooted second node from the 3 node GCS cluster.
3) Observing the pvc stuck at 38 for the time being.
4) Eventually the gd2 pods and csi nodeplugin came up and running, but the etcd remain in the same state.
5) Wait for 10 mins, the etcd pod gets killed and a new pod is spun
6) The cluster state is in healthy state.