Akka cluster integration

[krasserm]

Eventuate should run in an Akka cluster where each cluster node represents a separate Eventuate location. The integration has the following goals (prio :one:):

• EventsourcedActors can be replicated to all cluster nodes.
• EventsourcedActors are writable on all cluster nodes.
• EventsourcedActors remain writable during network partitions (split brain).
• Write conflicts can be resolved interactively or automatically (with existing API and tools).
• Adding/removing nodes to the cluster should dynamically add/remove these nodes to/from the Eventuate replication network.

Further goals are (prio :two:):

• Provide a client lib that allows for application-defined read consistency across multiple locations (eventual, causal, read-your-write, ...).
• Connect multiple Akka clusters e.g. running in different data centers via Eventuate replication connections.
• ...

With these building blocks in place, it should also be possible to implement features like:

• A cluster-wide persistent and reliable Akka EventBus with causal delivery semantics.
• A shard coordinator that works across multiple Akka clusters (in multiple data denters).
• ...

The initial focus is on prio :one: goals. Prerequisite for this ticket is #142.

[krasserm]

An initial implementation will use a LevelDB storage backend so that local writes to a cluster node are always possible. A shared Cassandra storage backend for all Akka cluster nodes might be in conflict with the requirement that EventsourcedActors should remain writable during network partitions. A hybrid solution, that uses LevelDB only if Cassandra is not available and re-sync later, could be an alternative.

On the other hand, applications with less strict durability requirements can still go for a LevelDB-only solution and recover lost events at one location from those already (asynchronously) replicated to other locations (see disaster recovery).

[hmatinho]

Hi Martin, is there a way to create the locations dynamically (and thus get the endpoint connections) from the the Akka Cluster seed nodes ?

[krasserm]

@hmatinho that's covered with #142. I plan to start working on it in a few weeks.

[calvinlfer]

It would be great if Eventuate would be able to work with Akka Cluster Sharding so we can make use of their model and when a network partition happens, it would continue to work under the conditions you described.

Update After looking at this again with some more Eventuate knowledge, Akka Cluster does not provide automatic recovery (for partitions) like Eventuate does with its Replication model so in the case of a network partition, there would have to be manual intervention instead of the nodes automatically recovering unless you have SBR and use something like ConductR which costs $$$. Really, the sharding model that Akka Cluster Sharding provides is the only behavior needed which ensures only one actor with a unique ID is present in the cluster. In the case of a network partition, two or more may be created but once the partition has been resolved (automatically), then conflict resolution would occur and one actor with that unique ID would be present again with merged updates

[coding4m]

@krasserm hi Martin, this feature will be released in the v-0.8?

[krasserm]

@coding4m no, most likely not.

[Andrei-Pozolotin]

+1