We're encountering an odd case where a Vault leader is being replaced by a single follower that is restarted in a cluster of 3. I'm running vault 1.15.4 running on GKE and deployed by helm version 0.26.1.
The leader was vault-0, I deleted one of the followers vault-1, and once vault-1 pod was created it became the current leader.
I then deleted vault-2 (follower), and pod vault-0 became the leader.
I deleted the current leader vault-0 and the new vault-1. became the leader (expected behavior)
I was expecting a real HA scenario where on steps 1 and 2 (follower deletion), the leader should continue to be the same without triggering the leader election.
vault operator raft autopilot get-config
Key Value
--- -----
Cleanup Dead Servers false
Last Contact Threshold 10s
Dead Server Last Contact Threshold 24h0m0s
Server Stabilization Time 10s
Min Quorum 0
Max Trailing Logs 1000
Disable Upgrade Migration false
We're encountering an odd case where a Vault leader is being replaced by a single follower that is restarted in a cluster of 3. I'm running vault 1.15.4 running on GKE and deployed by helm version 0.26.1.
I was expecting a real HA scenario where on steps 1 and 2 (follower deletion), the leader should continue to be the same without triggering the leader election.
Here's my config:
The autopilot config:
And probes:
Seems to be related https://github.com/hashicorp/vault/issues/14153