Cluster Singleton improvements

[rogeralsing]

It's easy to have a cluster singleton, you simply use a fixed single identity for your singleton and whenever someone passes messages to that identity, it will come alive. The standard virtual actor machinery.

But. what if you don't send a message to it? if you want to guarantee it exists, because it needs to do work on its own.

We could introduce the concept of singletons in the cluster config. Whenever a singleton is activated, it registers itself in the cluster gossip state. Then. all nodes that knows that this singleton should exist. they scan for it's existence in the cluster gossip state upon gossip events.

If none of the nodes has the singleton, then anyone that detects this, could just send a Touch event to awaken the singleton. This will register it in the gossip state and other nodes will know about it. No risk of duplicates as its handled by the standard identity lookup.

if it dies, the owner node removes it from the gossip state, and everyone will detect this

[rogeralsing]

We could probably abstract this in such a way that one could Track and Untrack identities this way, via code.

e.g. a manager actor calls Track for each identity of multiple workflows. Now the cluster will know that those workflow actors must exist, if not, trigger them to exist

[rogeralsing]

We need some way for these singleton actors to be registered into the gossip state somehow. Either they do it themselves upon start / stop. Or, the activator actor for the different providers do this.

This is needed so that the singleton monitor that watches the gossip state can see if the actor exists or not.

What would be the best approach here? cc @mhelleborg

[rogeralsing]

Maybe just publish Started and Stopped events on the eventstream from the activator. then the monitor can subscribe there and check if the actor is part of the known singletons

[rogeralsing]

I'm thinking we can have a middleware that handles lifecycle start and stop, e.g. Props.WithSingletonRegistration This will add and remove the entry from the gossip.