apple / swift-cluster-membership

Distributed Membership Protocol implementations in Swift
https://apple.github.io/swift-cluster-membership/
Apache License 2.0
201 stars 20 forks source link

+swim adds ability to handle "replacements" of nodes (new UID on same host/port) #68

Closed ktoso closed 4 years ago

ktoso commented 4 years ago

Turns out we can't ignore this case in SWIM Instance itself, and it should be handled natively here as well, in order to mark nodes dead and remove them from the members/all after all.

Motivation:

In stable node setup environments where instances can die and be revived on same host/port but get new UIDs when they do, we would not properly handle such a "replacement" coming in on the SWIM layer.

A replacement means that the "old instance on the same host/port is definitely dead, and the new one replaces it" as such an addMember may have to take into account declaring a node dead. This immediately causes .dead and does not care about unreachability.

Modifications:

Result:

ktoso commented 4 years ago

Additional follow up to tombstone dead members which is important in relation to such "same node" situations, esp. so we ensure we dont keep gossiping to a node which we thought we had removed

ktoso commented 4 years ago

Added a specific check and rejecting such noUID members as well as some docs in https://github.com/apple/swift-cluster-membership/pull/68/commits/ba0dc77b61a051a758b21b4949fc658307678729

Thanks for the feedback, was a good check to add 👍