search

camunda / camunda

Process Orchestration Framework
https://camunda.com/platform/
3.27k stars 594 forks source link

Follower joins some time triggers a new election #4147

Closed deepthidevaki closed 4 years ago

deepthidevaki commented 4 years ago

Expected behavior

When a follower joins the cluster there is no new election if there is already a leader.

Actual behavior

To distributed the leaders across the cluster uniformly, we predefine the leaders for each partition. On bootstrap we promote these predefined leaders to candidate which triggers an election even if there is already a leader. If this node doesn't have the up-to-date log it cannot win the election. So it won't lead to an inconsistent state. However, the election cause leader changes unnecessarily.

Log 
=== Bootstrap
6:24:51.515 [] [raft-server-2-raft-partition-partition-1] INFO  io.atomix.protocols.raft.roles.FollowerRole - RaftServer{raft-partition-partition-1}{role=FOLLOWER} - No heartbeat from null in the last PT3.122S (calculated from last 3178 ms), sending poll requests
16:24:51.516 [] [raft-server-2-raft-partition-partition-1] INFO  io.atomix.protocols.raft.impl.RaftContext - RaftServer{raft-partition-partition-1} - Transitioning to CANDIDATE
16:24:51.517 [] [raft-server-2-raft-partition-partition-1] INFO  io.atomix.protocols.raft.roles.CandidateRole - RaftServer{raft-partition-partition-1}{role=CANDIDATE} - Starting election
16:24:51.534 [] [raft-server-2-raft-partition-partition-1] INFO  io.atomix.protocols.raft.impl.RaftContext - RaftServer{raft-partition-partition-1} - Transitioning to LEADER
16:24:51.534 [] [raft-server-2-raft-partition-partition-1] INFO  io.atomix.protocols.raft.impl.RaftContext - RaftServer{raft-partition-partition-1} - Found leader 2
16:24:51.616 [] [raft-server-1-raft-partition-partition-1] INFO  io.atomix.protocols.raft.impl.RaftContext - RaftServer{raft-partition-partition-1} - Found leader 2
16:24:51.616 [] [raft-server-0-raft-partition-partition-1] INFO  io.atomix.protocols.raft.impl.RaftContext - RaftServer{raft-partition-partition-1} - Found leader 2
16:24:51.723 [] [raft-partition-group-raft-partition-0] INFO  io.atomix.protocols.raft.partition.RaftPartitionGroup - Started

== Stop Node 0
16:24:55.644 [] [raft-server-0-raft-partition-partition-1] INFO  io.atomix.protocols.raft.impl.RaftContext - RaftServer{raft-partition-partition-1} - Transitioning to INACTIVE

== Restart Node 0
16:25:01.478 [] [raft-server-0-raft-partition-partition-1] INFO  io.atomix.protocols.raft.impl.RaftContext - RaftServer{raft-partition-partition-1} - Transitioning to CANDIDATE
16:25:01.515 [] [raft-server-0-raft-partition-partition-1] INFO  io.atomix.protocols.raft.roles.CandidateRole - RaftServer{raft-partition-partition-1}{role=CANDIDATE} - Starting election
16:25:01.529 [] [raft-server-2-raft-partition-partition-1] INFO  io.atomix.protocols.raft.roles.LeaderRole - RaftServer{raft-partition-partition-1}{role=LEADER} - Received greater term from 0
16:25:01.529 [] [raft-server-2-raft-partition-partition-1] INFO  io.atomix.protocols.raft.impl.RaftContext - RaftServer{raft-partition-partition-1} - Transitioning to FOLLOWER
16:25:01.530 [] [raft-server-0-raft-partition-partition-1] INFO  io.atomix.protocols.raft.impl.RaftContext - RaftServer{raft-partition-partition-1} - Transitioning to FOLLOWER

16:25:04.915 [] [raft-server-0-raft-partition-partition-1] INFO  io.atomix.protocols.raft.roles.FollowerRole - RaftServer{raft-partition-partition-1}{role=FOLLOWER} - No heartbeat from null in the last PT3.379S (calculated from last 3385 ms), sending poll requests
16:25:05.466 [] [raft-server-1-raft-partition-partition-1] INFO  io.atomix.protocols.raft.roles.FollowerRole - RaftServer{raft-partition-partition-1}{role=FOLLOWER} - No heartbeat from null in the last PT4.096S (calculated from last 4096 ms), sending poll requests
16:25:05.515 [] [raft-server-2-raft-partition-partition-1] INFO  io.atomix.protocols.raft.roles.FollowerRole - RaftServer{raft-partition-partition-1}{role=FOLLOWER} - Accepted PollRequest{term=2, candidate=1, lastLogIndex=5, lastLogTerm=1}: candidate's log is up-to-date
16:25:05.515 [] [raft-server-0-raft-partition-partition-1] INFO  io.atomix.protocols.raft.roles.FollowerRole - RaftServer{raft-partition-partition-1}{role=FOLLOWER} - Accepted PollRequest{term=2, candidate=1, lastLogIndex=5, lastLogTerm=1}: candidate's log is up-to-date
16:25:05.515 [] [raft-server-2-raft-partition-partition-1] INFO  io.atomix.protocols.raft.roles.FollowerRole - RaftServer{raft-partition-partition-1}{role=FOLLOWER} - No heartbeat from null in the last PT3.979S (calculated from last 3986 ms), sending poll requests
16:25:05.515 [] [raft-server-1-raft-partition-partition-1] INFO  io.atomix.protocols.raft.impl.RaftContext - RaftServer{raft-partition-partition-1} - Transitioning to CANDIDATE
16:25:05.515 [] [raft-server-0-raft-partition-partition-1] INFO  io.atomix.protocols.raft.roles.FollowerRole - RaftServer{raft-partition-partition-1}{role=FOLLOWER} - Accepted PollRequest{term=2, candidate=2, lastLogIndex=5, lastLogTerm=1}: candidate's log is up-to-date
16:25:05.516 [] [raft-server-1-raft-partition-partition-1] INFO  io.atomix.protocols.raft.roles.CandidateRole - RaftServer{raft-partition-partition-1}{role=CANDIDATE} - Starting election
16:25:05.516 [] [raft-server-2-raft-partition-partition-1] INFO  io.atomix.protocols.raft.impl.RaftContext - RaftServer{raft-partition-partition-1} - Transitioning to CANDIDATE
16:25:05.516 [] [raft-server-2-raft-partition-partition-1] INFO  io.atomix.protocols.raft.roles.CandidateRole - RaftServer{raft-partition-partition-1}{role=CANDIDATE} - Starting election

Steps to reproduce

Create a cluster with 1 partition and replication factor 3 wait until the leaders are elected. If broker 0 is not the leader, restart it and then you can see the above behavior.

I will try to come up with code to reproduce it.

Zelldon commented 4 years ago

I think we need to take a look at it since it is cause of several flaky tests. Maybe I have time tomorrow