search

hashicorp / memberlist

Golang package for gossip based membership and failure detection
Mozilla Public License 2.0
3.61k stars 435 forks source link

memberlist fails in large-scale clusters #299

Open zhuohuashiyi opened 5 months ago

zhuohuashiyi commented 5 months ago

We recently implemented a large scale improved gossip protocol based on memberlist. memberlist did not meet our expectations when the number of nodes reached 1000. After investigation, it is found that the broadcast mechanism takes data from the system broadcast queue first(function getBraodcasts in broadcast.go). If the number of nodes is too large, the UDP packet does not have enough space to store the data in the user-defined broadcast queue, which leads to our failure to achieve system consistency.

zhuohuashiyi commented 5 months ago

No, I conducted experiments based on memberlist in a cluster consisting of thousands of servers. The real cause of the problem I mentioned is that the udp packet size of a single broadcast is limited, and memberlist's broadcast logic is to first populate the system data, that is, some node status information, and then fetch data from the user-defined broadcast queue to populate the udp packet. This logic is fine in a small cluster, but if the cluster scale increases, udp packets will be full of node status information, resulting in inconsistency at the user level.