search

gluster / gcs

Check github.com/heketi, github.com/gluster/gluster-containers, or github.com/kadalu/kadalu as active alternatives
https://gluster.org
Apache License 2.0
36 stars 24 forks source link

etcd pod in Error state for 10 mins after a node reboot #95

Open ksandha opened 5 years ago

ksandha commented 5 years ago

1) Created 50 PVC parallel requests. 2) While the pvc are getting processed , rebooted second node from the 3 node GCS cluster.

[vagrant@kube1 ~]$ kubectl get pods -n gcs
NAME                                   READY   STATUS    RESTARTS   AGE
csi-attacher-glusterfsplugin-0         2/2     Running   0          12m
csi-nodeplugin-glusterfsplugin-69jvq   2/2     Running   0          12m
csi-nodeplugin-glusterfsplugin-bf5kx   2/2     Running   0          12m
csi-nodeplugin-glusterfsplugin-n7wzw   0/2     Error     0          12m
csi-provisioner-glusterfsplugin-0      3/3     Running   0          12m
etcd-9dnxxnvhq2                        1/1     Running   0          18m
etcd-9wz7txxx59                        1/1     Running   0          16m
etcd-b92cpssvmz                        0/1     Error     0          17m
etcd-operator-7cb5bd459b-mczrk         1/1     Running   0          18m
gluster-kube1-0                        1/1     Running   1          16m
gluster-kube2-0                        0/1     Error     1          16m
gluster-kube3-0                        1/1     Running   1          16m

3) Observing the pvc stuck at 38 for the time being. 

[vagrant@kube1 ~]$ kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM               STORAGECLASS    REASON   AGE
pvc-d92e016d-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc0    glusterfs-csi            116s
pvc-d959ddbe-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc1    glusterfs-csi            115s
pvc-d9860619-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc2    glusterfs-csi            2m19s
pvc-d9b5e137-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc3    glusterfs-csi            2m19s
pvc-da1ac34c-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc5    glusterfs-csi            2m18s
pvc-da510fc2-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc6    glusterfs-csi            2m18s
pvc-db002280-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc9    glusterfs-csi            2m17s
pvc-db342a77-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc10   glusterfs-csi            2m15s
pvc-db66d2fe-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc11   glusterfs-csi            119s
pvc-db9bcff5-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc12   glusterfs-csi            2m16s
pvc-dbce239d-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc13   glusterfs-csi            114s
pvc-dc04ef7a-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc14   glusterfs-csi            2m15s
pvc-dc39d4bc-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc15   glusterfs-csi            2m14s
pvc-dca4b450-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc17   glusterfs-csi            2m13s
pvc-dcddb56f-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc18   glusterfs-csi            2m12s
pvc-dd136269-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc19   glusterfs-csi            2m12s
pvc-dd459ffd-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc20   glusterfs-csi            118s
pvc-ddaed68a-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc22   glusterfs-csi            116s
pvc-dde6b8bb-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc23   glusterfs-csi            2m11s
pvc-de4d84fc-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc25   glusterfs-csi            2m9s
pvc-de8758b5-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc26   glusterfs-csi            119s
pvc-dec2600c-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc27   glusterfs-csi            2m9s
pvc-def3a474-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc28   glusterfs-csi            2m9s
pvc-df2e25d9-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc29   glusterfs-csi            2m8s
pvc-df64fa77-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc30   glusterfs-csi            2m6s
pvc-df99124a-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc31   glusterfs-csi            2m6s
pvc-dfce5481-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc32   glusterfs-csi            2m5s
pvc-e0066f0b-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc33   glusterfs-csi            2m4s
pvc-e0431efb-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc34   glusterfs-csi            2m3s
pvc-e0925da6-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc35   glusterfs-csi            2m3s
pvc-e0c314f3-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc36   glusterfs-csi            2m2s
pvc-e0f896db-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc37   glusterfs-csi            2m1s
pvc-e12be8b5-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc38   glusterfs-csi            119s

4) Eventually the gd2 pods and csi nodeplugin came up and running, but the etcd remain in the same state.

[vagrant@kube1 ~]$ kubectl get pods -n gcs
NAME                                   READY   STATUS    RESTARTS   AGE
csi-attacher-glusterfsplugin-0         2/2     Running   0          15m
csi-nodeplugin-glusterfsplugin-69jvq   2/2     Running   0          15m
csi-nodeplugin-glusterfsplugin-bf5kx   2/2     Running   0          15m
csi-nodeplugin-glusterfsplugin-n7wzw   2/2     Running   2          15m
csi-provisioner-glusterfsplugin-0      3/3     Running   0          15m
etcd-9dnxxnvhq2                        1/1     Running   0          21m
etcd-9wz7txxx59                        1/1     Running   0          19m
etcd-b92cpssvmz                        0/1     Error     0          20m
etcd-operator-7cb5bd459b-mczrk         1/1     Running   0          21m
gluster-kube1-0                        1/1     Running   1          19m
gluster-kube2-0                        1/1     Running   3          19m
gluster-kube3-0                        1/1     Running   1          19m

5) Wait for 10 mins, the etcd pod gets killed and a new pod is spun 

[vagrant@kube1 ~]$ kubectl get pods -n gcs
NAME                                   READY   STATUS     RESTARTS   AGE
csi-attacher-glusterfsplugin-0         2/2     Running    0          22m
csi-nodeplugin-glusterfsplugin-69jvq   2/2     Running    0          22m
csi-nodeplugin-glusterfsplugin-bf5kx   2/2     Running    0          22m
csi-nodeplugin-glusterfsplugin-n7wzw   2/2     Running    2          22m
csi-provisioner-glusterfsplugin-0      3/3     Running    0          22m
etcd-9dnxxnvhq2                        1/1     Running    0          28m
etcd-9wz7txxx59                        1/1     Running    0          26m
etcd-dv6q2n7c27                        0/1     Init:0/1   0          26s
etcd-operator-7cb5bd459b-mczrk         1/1     Running    0          28m
gluster-kube1-0                        1/1     Running    1          26m
gluster-kube2-0                        1/1     Running    3          26m
gluster-kube3-0                        1/1     Running    1          26m
[vagrant@kube1 ~]$

6) The cluster state is in healthy state.

[vagrant@kube1 ~]$ kubectl get pods -n gcs 
NAME                                   READY   STATUS    RESTARTS   AGE
csi-attacher-glusterfsplugin-0         2/2     Running   0          22m
csi-nodeplugin-glusterfsplugin-69jvq   2/2     Running   0          22m
csi-nodeplugin-glusterfsplugin-bf5kx   2/2     Running   0          22m
csi-nodeplugin-glusterfsplugin-n7wzw   2/2     Running   2          22m
csi-provisioner-glusterfsplugin-0      3/3     Running   0          22m
etcd-9dnxxnvhq2                        1/1     Running   0          28m
etcd-9wz7txxx59                        1/1     Running   0          27m
etcd-dv6q2n7c27                        1/1     Running   0          54s
etcd-operator-7cb5bd459b-mczrk         1/1     Running   0          29m
gluster-kube1-0                        1/1     Running   1          27m
gluster-kube2-0                        1/1     Running   3          27m
gluster-kube3-0                        1/1     Running   1          27m
[vagrant@kube1 ~]$ 
[vagrant@kube1 ~]$ 
[vagrant@kube1 ~]$ 
[vagrant@kube1 ~]$ 
[vagrant@kube1 ~]$ 
[vagrant@kube1 ~]$ kubectl get pv
NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM               STORAGECLASS    REASON   AGE
pvc-d92e016d-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc0    glusterfs-csi            11m
pvc-d959ddbe-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc1    glusterfs-csi            11m
pvc-d9860619-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc2    glusterfs-csi            11m
pvc-d9b5e137-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc3    glusterfs-csi            11m
pvc-d9ec5839-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc4    glusterfs-csi            80s
pvc-da1ac34c-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc5    glusterfs-csi            11m
pvc-da510fc2-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc6    glusterfs-csi            11m
pvc-da95c9c6-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc7    glusterfs-csi            76s
pvc-db002280-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc9    glusterfs-csi            11m
pvc-db342a77-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc10   glusterfs-csi            11m
pvc-db66d2fe-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc11   glusterfs-csi            11m
pvc-db9bcff5-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc12   glusterfs-csi            11m
pvc-dbce239d-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc13   glusterfs-csi            11m
pvc-dc04ef7a-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc14   glusterfs-csi            11m
pvc-dc39d4bc-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc15   glusterfs-csi            11m
pvc-dc6dc420-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc16   glusterfs-csi            64s
pvc-dca4b450-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc17   glusterfs-csi            11m
pvc-dcddb56f-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc18   glusterfs-csi            11m
pvc-dd136269-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc19   glusterfs-csi            11m
pvc-dd459ffd-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc20   glusterfs-csi            11m
pvc-dd7b42c5-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc21   glusterfs-csi            77s
pvc-ddaed68a-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc22   glusterfs-csi            11m
pvc-dde6b8bb-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc23   glusterfs-csi            11m
pvc-de1a6f0f-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc24   glusterfs-csi            80s
pvc-de4d84fc-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc25   glusterfs-csi            11m
pvc-de8758b5-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc26   glusterfs-csi            11m
pvc-dec2600c-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc27   glusterfs-csi            11m
pvc-def3a474-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc28   glusterfs-csi            11m
pvc-df2e25d9-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc29   glusterfs-csi            11m
pvc-df64fa77-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc30   glusterfs-csi            11m
pvc-df99124a-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc31   glusterfs-csi            11m
pvc-dfce5481-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc32   glusterfs-csi            11m
pvc-e0066f0b-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc33   glusterfs-csi            11m
pvc-e0431efb-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc34   glusterfs-csi            11m
pvc-e0925da6-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc35   glusterfs-csi            11m
pvc-e0c314f3-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc36   glusterfs-csi            11m
pvc-e0f896db-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc37   glusterfs-csi            11m
pvc-e12be8b5-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc38   glusterfs-csi            11m
pvc-e1610f73-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc39   glusterfs-csi            85s
pvc-e1cecca8-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc41   glusterfs-csi            85s
pvc-e2072dd3-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc42   glusterfs-csi            84s
pvc-e23c35fa-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc43   glusterfs-csi            82s
pvc-e2764934-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc44   glusterfs-csi            79s
pvc-e2b1a377-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc45   glusterfs-csi            82s
pvc-e2eefa00-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc46   glusterfs-csi            81s
pvc-e326ff76-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc47   glusterfs-csi            80s
pvc-e35b14fa-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc48   glusterfs-csi            78s
pvc-e39289e9-0e6a-11e9-a7a5-525400ee92d8   2Gi        RWX            Delete           Bound    default/gcs-pvc49   glusterfs-csi            78s
ksandha commented 5 years ago

etcd-9dnxxnvhq2.log etcd-9wz7txxx59.log etcd-dv6q2n7c27.log

ksandha commented 5 years ago

rhsqa-virt05.lab.eng.blr.redhat.com:/root/isse95_gd2.log glusterd2 logs placed at above location