search

akka / akka-projection

Akka Projections is intended for building systems with the CQRS pattern, and facilitate in event-based service-to-service communication.
https://doc.akka.io/libraries/akka-projection/current/
Other
102 stars 35 forks source link

Share a sharded daemon between multiple projections #171

Open jroper opened 4 years ago

jroper commented 4 years ago

You may not be using sharded event streams, but you may have many entity types, or many projections on the same entity, and putting the load of all those on the same node (which is what happens when you use a cluster singleton for each) may not be desirable.

This is might just be a documentation task (though some tooling to help would be useful), but I think many users would find it useful to be able to use the sharded daemon to run many projections, and balance projections across the cluster, not just projection tags. With the current API, this would require placing all your projection behaviors into an IndexedSeq (and for sharded projections, all the shards of each behavior), which can be looked up by the callback that returns the behaviors. While not that difficult, a I think a Builder abstraction and allowed you to add multiple tasks to it (two add methods, one for single tasks, one for sharded tasks), and then pass that to ShardedDaemonProcess, would make for much more readable user code, as it would make clearer the intent. Perhaps such an API belongs in Akka core, but I'm raising it here because the use case is for projections.

patriknw commented 4 years ago

You may not be using sharded event streams, but you may have many entity types, or many projections on the same entity, and putting the load of all those on the same node (which is what happens when you use a cluster singleton for each) may not be desirable.

You can still run such Projections with ShardedDaemonProcess instead of Cluster Singleton. A ShardedDaemonProcess can have just 1 instance.

Maybe LeastShardAllocationStrategy in Cluster Sharding will currently pick the same node for the first allocateShard meaning that different entity types will end up at the same node. That is something we can change in Akka. Making it random.

Sure, there could be a few other minor advantages of running different types of projections with the same ShardedDaemonProcess. That is already possible, but maybe not obvious or convenient. Let's look into that when we decide if we need a more specific ProjectionDaemonProcess API, which is discussed in https://github.com/akka/akka-projection/issues/137