When merging gossips, also potentially apply/emit reachability change.

[ktoso]

We should consider this:

• SWIM is distributed and spreads suspicion and unreachability
  • [ ] TODO: check if we spread unreachability
• we gossip Members in high-level, they include reachability
  • we know if a gossip is "ahead" and thus can pick the "more recent information about unreachability
  • though that means we declared unreachability WITHOUT the SWIM declaring so locally, is this weird? Should we JUST rely on SWIMs to tell eachother the unreachability?

I would like to discuss this one some more, rather than just do what "we're used to" -- as our more smart failure detector does the spreading the info, so perhaps we should NOT merge this information on the high-level?

[ktoso]

Today:

we do NOT merge unreachability in high level, we ignore it and rely on local FD to detect these states.

[ktoso]

I wrote some tests but didn't include them as it's a bigger conceptual change (or no change):

    func test_mergeForward_reachability_shouldBeTakenFromMoreRecentIncomingObservation_sameStatus() {
        var membership = Cluster.Membership.parse(
            """
            A.up B.up
            """, nodes: self.allNodes
        )

        var ahead = Cluster.Membership.parse(
            """
            A.up B.up 
            """, nodes: self.allNodes
        )
        ahead.mark(self.nodeA, reachability: .unreachable).shouldNotBeNil()

        let changes = membership.mergeFrom(incoming: ahead, myself: nil)

        changes.count.shouldEqual(0)
        changes.shouldEqual([
            // TODO we'd also want a Reachability change IFF we decide to merge ahead reachability status as well
            // no change to members, just reachability should change
        ])

        membership.shouldEqual(membership)
        membership.uniqueMember(self.nodeA)?.reachability.shouldEqual(.unreachable)
    }

    func test_mergeForward_reachability_shouldBeTakenFromMoreRecentIncomingObservation_diffStatus() {
        var membership = Cluster.Membership.parse(
            """
            A.joining B.up
            """, nodes: self.allNodes
        )

        var ahead = Cluster.Membership.parse(
            """
            A.up B.up 
            """, nodes: self.allNodes
        )
        ahead.mark(self.nodeA, reachability: .unreachable).shouldNotBeNil()

        let changes = membership.mergeFrom(incoming: ahead, myself: nil)

        // TODO: member should have right reachability and it'd be a change (!!!!)
        changes.count.shouldEqual(1)
        changes.shouldEqual([
            // TODO we'd also want a Reachability change IFF we decide to merge ahead reachability status as well
            Cluster.MembershipChange(node: self.nodeA, fromStatus: .joining, toStatus: .up)
        ])

        membership.uniqueMember(self.nodeA)!.reachability.shouldEqual(.unreachable)
        membership.shouldEqual(membership)
    }