Implement nodes .leaving cluster allowing to perform graceful leaving

[ktoso]

E.g. wait for some thing to hand over work to other nodes etc.

Examples that should use this:

• singletons
• virtual actors
• anything the user wants to complete before shutting down gracefully

[akbashev]

@ktoso how third point will work? System will notify user that node is about to leave?

[ktoso]

When you look at cluster events here: https://apple.github.io/swift-distributed-actors/1.0.0-beta.2/documentation/distributedactors/clustering there's a "leaving" state, we don't make much use of it.

It can be issued when the node is leaving "gracefully" in that event, we could first send a "leaving self" event, give some subsystems time to perform a "hand over (e.g. the singleton to another node") before declaring the self as down.

This matters because once a system is declared down and other nodes hear about this, they will reject messages from it. So such handover may not complete successfully. In other words, the leaving event always eventually results in "down", but it should give the system some more time to hand over events, flush in memory state etc.

Systems like singleton etc would, as usual (and as they do now), listen on cluster events, and initiate the hand over (or whatever they want to do, when they see the leaving state. User code can do the same. Users can also ignore the leaving event and only handle "down" as a leaving always moves to down eventually.

[akbashev]

@ktoso thx for clearing up, sounds promising!