Closed mhristof closed 1 year ago
@mhristof Thanks for filing this ticket. I am trying to understand the issue better. Would you please elaborate more on your setup? Are these Vault servers running in the same host? I see that one node is unhealthy, and it is not the leader node. So, I am wondering to which server the health request is sent? As mentioned in sys/health docs, 200 will be returned only from the active (leader) node. But, it is not clear to which server the health request is sent from the output snippet. Besides, would you please describe how many servers are being removed/added from/to the cluster every 10 minutes? What happens to the active node in that scenario? Is it also being replaced with another node?
Hello @mhristof - since it's been a while since we've heard from you on this issue, I'm going to go ahead and close it for now. Please feel free to re-open it or open a new issue as needed. Thanks!
Describe the bug a vault node reports 200 in
sys/health
and the rest of the cluster has removed it since it was reported as raft unhealthy.To Reproduce Steps to reproduce the behavior:
vault operator raft autopilot set-config \ -cleanup-dead-servers=true \ -min-quorum=3 \ -dead-server-last-contact-threshold=20s \ -max-trailing-logs=5000
/v1/sys/health
endpoint and status codes200,429
Expected behavior
sys/health
reports anything but 200 so the node can be removed from the autoscaling groupEnvironment:
vault status
):From an external node checking the cluster:
vault version
):Vault server configuration file(s):
each node as
node_id
the AWS instance id, ffor exampleAdditional context From the faulty node, i can see