search

habitat-sh / habitat

Modern applications with built-in automation
https://www.habitat.sh
Apache License 2.0
2.6k stars 314 forks source link

Election: Don't send election messages without quorum #5893

Open baumanj opened 5 years ago

baumanj commented 5 years ago

In elections we have the concept of quorum as defined by the butterfly server's check_quorum method. Functionally, it means that enough of the supervisor nodes we know about are healthy such that we can have an election without fear of getting into a split-brain scenario.

However, because quorum is based on the membership protocol and the membership protocol is subject to seeing other nodes as unhealthy due to network congestion, it can be hard to complete an election when the network is congested. Unfortunately adding the election messages into the mix makes congestion worse.

Currently, an election can't finish without quorum, but otherwise the lack of quorum has no effect on how elections run. To avoid unnecessary work that may worsen congestion, we should investigate the possibility of only initiating (and possibly propagating) election messages when there is quorum.

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. We value your input and contribution. Please leave a comment if this issue still affects you.

stale[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. We value your input and contribution. Please leave a comment if this issue still affects you.

stale[bot] commented 11 months ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. We value your input and contribution. Please leave a comment if this issue still affects you.