Remote State auto restore trigger when all local disk state is lost, i.e. all cluster manager node are replaced
While restoring Remote State when new nodes comeup, we restore the decommission attributes but not the VotingConfiguration.
The decommissioned node will take part in election of new cluster manager since we did not restore the VotingConfiguration and new VotingConfiguration construction will allow the decommissioned node to take part in preVoting round.
If the election is successful by a vote from the decommissioned node, the first publish will fail as the decommissioned node vote would be considered invalid because the decommissioned node wont be able to join the elected cluster manager.
This will fail the elected cluster manager and another eligible cluster manager will start election.
The new election again can fail if the decommissioned node again votes.
The decommission node will not ever start election as we already have a check for nodeCommissioned in this flow.
To Reproduce
Steps to reproduce the behavior:
Create a 3 dedicated cluster manager remote store and remote state domain with force zone awareness enabled.
Weight out a dedicated cluster manager node using a node attr
Decommission the weighed out cluster manager node.
Replace all cluster manager nodes
The new cluster manager nodes will download the remote state with decommission attributes
Cluster would go into election loops
Expected behavior
Decommissioned node should not participate in voting
Describe the bug
To Reproduce Steps to reproduce the behavior:
Expected behavior Decommissioned node should not participate in voting