search

hazelcast / hazelcast

Hazelcast is a unified real-time data platform combining stream processing with a fast data store, allowing customers to act instantly on data-in-motion for real-time insights.
https://www.hazelcast.com
Other
6.09k stars 1.83k forks source link

Split Brain Condition does not get merged in kubernetes environment with 4.2.1 although nodes seem to agree. #19144

Open abelmannu opened 3 years ago

abelmannu commented 3 years ago

We are using hazelcast 4.2.1 in an kubernetes environment with openjdk:14-jdk-slim images. In our dev environment, where we only have two nodes, these two nodes sometimes (short after every 5th deployment) end up having a split brain condition and do not merge, although they find each other and agree on what to do:

The joiner of the first nodes says the second node should join. And the joiner of the second not it should join the first node. But nothing happens. The log repeats every couple of minutes and the clusters are not merged.

It does not matter if we use a merge policy or not. More often than not it works without any problems.

Log of first node:

2021-07-20 09:14:08.306 DEBUG 142 --- [hz.hazelcast-instance.cached.thread-4] c.h.i.cluster.impl.MembershipManager     : [10.41.31.101]:5701 [light-cluster] [4.2.1] Sending member list to the non-master nodes:

Members {size:1, ver:5} [
        Member [10.41.31.101]:5701 - 7263bccd-f330-4b96-8b52-f22db7c7a90e this
]

2021-07-20 09:14:08.446 DEBUG 142 --- [hz.hazelcast-instance.cached.thread-5] c.h.i.cluster.impl.DiscoveryJoiner       : [10.41.31.101]:5701 [light-cluster] [4.2.1] Sending SplitBrainJoinMessage to [10.41.31.102]:5701
2021-07-20 09:14:08.448 DEBUG 142 --- [hz.hazelcast-instance.cached.thread-5] c.h.i.cluster.impl.ClusterJoinManager    : [10.41.31.101]:5701 [light-cluster] [4.2.1] Checking if we should merge to: SplitBrainJoinMessage{packetVersion=4, buildNumber=20210630, memberVersion=4.2.1, clusterVersion=4.2, address=[10.41.31.102]:5701, uuid='9cdd64b4-62c8-4f19-bf29-d3cef4e8e2f6', liteMember=false, memberCount=1, dataMemberCount=1, memberListVersion=1}
2021-07-20 09:14:08.449  INFO 142 --- [hz.hazelcast-instance.cached.thread-5] c.h.i.cluster.impl.ClusterJoinManager    : [10.41.31.101]:5701 [light-cluster] [4.2.1] [10.41.31.102]:5701 should merge to us, both have the same data member count: 1
2021-07-20 09:14:23.277 DEBUG 142 --- [hz.hazelcast-instance.cached.thread-4] c.h.i.p.InternalPartitionService         : [10.41.31.101]:5701 [light-cluster] [4.2.1] Checking partition state, stamp: -5900145379368197006

Log of second node:


2021-07-20 09:14:24.149 DEBUG 141 --- [hz.hazelcast-instance.cached.thread-4] c.h.i.p.InternalPartitionService         : [10.41.31.102]:5701 [light-cluster] [4.2.1] Checking partition state, stamp: -8661523421455686299
2021-07-20 09:14:24.175 DEBUG 141 --- [hz.hazelcast-instance.cached.thread-4] c.h.s.d.integration.DiscoveryService     : [10.41.31.102]:5701 [light-cluster] [4.2.1] Using service name to discover nodes.
2021-07-20 09:14:24.176 DEBUG 141 --- [hz.hazelcast-instance.cached.thread-6] c.h.i.cluster.impl.MembershipManager     : [10.41.31.102]:5701 [light-cluster] [4.2.1] Sending member list to the non-master nodes:

Members {size:1, ver:1} [
        Member [10.41.31.102]:5701 - 9cdd64b4-62c8-4f19-bf29-d3cef4e8e2f6 this
]

2021-07-20 09:14:39.149 DEBUG 141 --- [hz.hazelcast-instance.cached.thread-4] c.h.i.p.InternalPartitionService         : [10.41.31.102]:5701 [light-cluster] [4.2.1] Checking partition state, stamp: -8661523421455686299
2021-07-20 09:14:54.148 DEBUG 141 --- [hz.hazelcast-instance.cached.thread-6] c.h.i.p.InternalPartitionService         : [10.41.31.102]:5701 [light-cluster] [4.2.1] Checking partition state, stamp: -8661523421455686299
2021-07-20 09:15:08.423 DEBUG 141 --- [hz.hazelcast-instance.priority-generic-operation.thread-0] c.h.i.cluster.impl.ClusterJoinManager    : [10.41.31.102]:5701 [light-cluster] [4.2.1] Checking if we should merge to: SplitBrainJoinMessage{packetVersion=4, buildNumber=20210630, memberVersion=4.2.1, clusterVersion=4.2, address=[10.41.31.101]:5701, uuid='7263bccd-f330-4b96-8b52-f22db7c7a90e', liteMember=false, memberCount=1, dataMemberCount=1, memberListVersion=5}
2021-07-20 09:15:08.423  INFO 141 --- [hz.hazelcast-instance.priority-generic-operation.thread-0] c.h.i.cluster.impl.ClusterJoinManager    : [10.41.31.102]:5701 [light-cluster] [4.2.1] We should merge to [10.41.31.101]:5701, both have the same data member count: 1
2021-07-20 09:15:08.424 DEBUG 141 --- [hz.hazelcast-instance.priority-generic-operation.thread-0] c.h.i.c.i.o.SplitBrainMergeValidationOp  : [10.41.31.102]:5701 [light-cluster] [4.2.1] Returning SplitBrainJoinMessage{packetVersion=4, buildNumber=20210630, memberVersion=4.2.1, clusterVersion=4.2, address=[10.41.31.102]:5701, uuid='9cdd64b4-62c8-4f19-bf29-d3cef4e8e2f6', liteMember=false, memberCount=1, dataMemberCount=1, memberListVersion=1} to [10.41.31.101]:5701
2021-07-20 09:15:09.148 DEBUG 141 --- [hz.hazelcast-instance.cached.thread-6] c.h.i.p.InternalPartitionService         : [10.41.31.102]:5701 [light-cluster] [4.2.1] Checking partition state, stamp: -8661523421455686299```
leszko commented 3 years ago

@abelmannu I could not reproduce the issue. Could you write the exact steps who to reproduce the Split Brain issue you encounter? Preferably with our official Helm Chart.

lukasherman commented 1 year ago

Just got into exactly this situation while running unrelated unit test complemented with standalone node joining another standalone node. Found out that in TcpIpJoiner#504

SplitBrainMergeCheckResult result = sendSplitBrainJoinMessageAndCheckResponse(address, request);
  if (result == SplitBrainMergeCheckResult.LOCAL_NODE_SHOULD_MERGE) {

result is always SplitBrainMergeCheckResult.REMOTE_NODE_SHOULD_MERGE, which is not handled by any node.

Split brain scenario is simulated by changing networkConfig. Changing first node config does not work, changing both node config does not work. Resolved by changing only second node networkConfig:

instance.getConfig().getNetworkConfig().getJoin().getTcpIpConfig().addMember("127.0.0.1:6702").addMember("127.0.0.1:6712");

Strangely, related unit test com.hazelcast.cluster.SplitBrainHandlerTest works well.

pavelorehov commented 1 year ago

HZ 5.1.5 with 2 nodes in embedded JVM, on kube environment, not activating CP safe mode.

Have exactly same issue occurs sometimes after pods restarts, nodes connected but waiting for each other, and none of them joining cluster.

24 00:41:32,159 [hz.laughing_ramanujan.priority-generic-operation.thread-0] INFO c.h.i.c.i.ClusterJoinManager - [fdfb:85ef:26ff:b454:c1:d7a7:37a7:3633]:5701 [aaa] [5.1.5] [fdfb:85ef:26ff:d7e6:e1cd:93ee:c50d:5a32]:5701 should merge to us, both have the same data member count: 1

24 00:55:46,842 [hz.fervent_moore.priority-generic-operation.thread-0] INFO c.h.i.c.i.ClusterJoinManager - [fdfb:85ef:26ff:d7e6:e1cd:93ee:c50d:5a32]:5701 [aaa] [5.1.5] We should merge to [fdfb:85ef:26ff:b454:c1:d7a7:37a7:3633]:5701, both have the same data member count: 1