Peer add to the node as a peer succeeded and the ONLINE status for the newly added shows as "no"

PrasadDesala commented 5 years ago

Peer add to the node as a peer succeeded and the ONLINE status for the newly added shows as "no".

Observed behavior

The below peer add IP is same as the node IP.

[root@gluster-kube2-0 /]# glustercli peer add 10.233.65.5 Peer add successful +--------------------------------------+-----------------+-------------------------------------+-------------------------------------+ | ID | NAME | CLIENT ADDRESSES | PEER ADDRESSES | +--------------------------------------+-----------------+-------------------------------------+-------------------------------------+ | c19971e4-946c-4cbd-b68a-7573af198ddd | gluster-kube2-0 | gluster-kube2-0.glusterd2.gcs:24007 | 10.233.65.5:24008 | | | | | gluster-kube2-0.glusterd2.gcs:24008 | +--------------------------------------+-----------------+-------------------------------------+-------------------------------------+

Expected/desired behavior

Peer add of the same node should fail.

Details on how to reproduce (minimal and precise)

1) Have a 3 node gcs setup (not in trusted pool yet) 2) from multiple nodes at a same time send peer add request of node2.

Information about the environment:

Glusterd2 version used (e.g. v4.1.0 or master): v6.0-dev.93.gitd4ca2cb
Operating system used: CentOS 7.5.1804
Glusterd2 compiled from sources, as a package (rpm/deb), or container: Container
Using External ETCD: (yes/no, if yes ETCD version): Yes version: 3.3.8
If container, which container image:
Using kubernetes, openshift, or direct install:
If kubernetes/openshift, is gluster running inside kubernetes/openshift or outside: Kubernetes

rishubhjain commented 5 years ago

@aravindavk This might be possible because of https://github.com/gluster/glusterd2/blob/master/glusterd2/commands/peers/addpeer.go#L58, this node might be having multiple addresses.

@PrasadDesala Can you print the output of glustercli peer status before adding the node? and can you paste the output of ip addr show?

PrasadDesala commented 5 years ago

kube3_glusterd2.log kube2_glusterd2.log kube1_glusterd2.log

PrasadDesala commented 5 years ago

@aravindavk This might be possible because of https://github.com/gluster/glusterd2/blob/master/glusterd2/commands/peers/addpeer.go#L58, this node might be having multiple addresses.

@PrasadDesala Can you print the output of glustercli peer status before adding the node? and can you paste the output of ip addr show?

root@gluster-kube2-0 /]# glustercli peer status +--------------------------------------+-----------------+-------------------------------------+-------------------------------------+--------+-----+ | ID | NAME | CLIENT ADDRESSES | PEER ADDRESSES | ONLINE | PID | +--------------------------------------+-----------------+-------------------------------------+-------------------------------------+--------+-----+ | 38859e8c-b279-4ce2-9b17-0a5bfaaaeeb0 | gluster-kube2-0 | gluster-kube2-0.glusterd2.gcs:24007 | gluster-kube2-0.glusterd2.gcs:24008 | yes | 25 | +--------------------------------------+-----------------+-------------------------------------+-------------------------------------+--------+-----+ [root@gluster-kube2-0 /]# glustercli peer add 10.233.65.5 Peer add successful +--------------------------------------+-----------------+-------------------------------------+-------------------------------------+ | ID | NAME | CLIENT ADDRESSES | PEER ADDRESSES | +--------------------------------------+-----------------+-------------------------------------+-------------------------------------+ | c19971e4-946c-4cbd-b68a-7573af198ddd | gluster-kube2-0 | gluster-kube2-0.glusterd2.gcs:24007 | 10.233.65.5:24008 | | | | | gluster-kube2-0.glusterd2.gcs:24008 | +--------------------------------------+-----------------+-------------------------------------+-------------------------------------+ [root@gluster-kube2-0 /]# glustercli peer status

[root@gluster-kube2-0 glusterd2]# ifconfig eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1450 inet 10.233.65.5 netmask 255.255.255.0 broadcast 0.0.0.0 inet6 fe80::d841:f2ff:feb4:2c62 prefixlen 64 scopeid 0x20 ether 0a:58:0a:e9:41:05 txqueuelen 0 (Ethernet) RX packets 308524 bytes 78159986 (74.5 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 328115 bytes 61065215 (58.2 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet 127.0.0.1 netmask 255.0.0.0 inet6 ::1 prefixlen 128 scopeid 0x10 loop txqueuelen 1000 (Local Loopback) RX packets 279504 bytes 41857567 (39.9 MiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 279504 bytes 41857567 (39.9 MiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

I think the machine has only one IP configured.

PrasadDesala commented 5 years ago

@rishubhjain As discussed I have reproduce this issue again. Once we end up in this situation we will not be able to remove that stale peer. Peer remove is failing with response body: peer is not alive. [root@gluster-kube3-0 /]# glustercli peer status +--------------------------------------+-----------------+-------------------------------------+-------------------------------------+--------+-----+ | ID | NAME | CLIENT ADDRESSES | PEER ADDRESSES | ONLINE | PID | +--------------------------------------+-----------------+-------------------------------------+-------------------------------------+--------+-----+ | 13852509-b5b7-4af4-9c2e-2a2b2e3d650f | gluster-kube3-0 | gluster-kube3-0.glusterd2.gcs:24007 | 10.233.66.6:24008 | yes | 25 | | | | | gluster-kube3-0.glusterd2.gcs:24008 | | | | e88b7161-5e75-4959-92d9-f569d100483d | gluster-kube3-0 | gluster-kube3-0.glusterd2.gcs:24007 | gluster-kube3-0.glusterd2.gcs:24008 | no | | +--------------------------------------+-----------------+-------------------------------------+-------------------------------------+--------+-----+ [root@gluster-kube3-0 /]# glustercli peer remove e88b7161-5e75-4959-92d9-f569d100483d Peer remove failed

Response headers: X-Gluster-Cluster-Id: 9e32dd64-28d1-44c8-a602-c9aa56fffd45 X-Gluster-Peer-Id: 13852509-b5b7-4af4-9c2e-2a2b2e3d650f X-Request-Id: bca719b3-dbf0-4693-87d2-cce4988820f8

Response body: peer is not alive

PrasadDesala commented 5 years ago

This issue is not always reproducible and seems to be some race.

gluster / glusterd2