search

Cratis / Chronicle

Event sourcing made easy—no complexity, just powerful tools and flexibility for everyone, from beginners to pros.
https://cratis.io
MIT License
32 stars 6 forks source link

Observer does not handle partitions separetely / parallel #1549

Open woksin opened 2 days ago

woksin commented 2 days ago

Because Observer is a grain which is not reentrant and the Handle method is also not reentrant it means that it will only handle one message at a time, a message being a list of events for a partition to handle. It will wait for the subscription to handle the events and then handle the state. A more performant way would be to completely parallelize and isolate the handling of the events and state for each partition separately

einari commented 1 day ago

It would be great if we could do this in a reliable manner. The challenge I see is that we have state that is related to the observer that would be hard to maintain in a reentrant model. We have today the following properties that need to be correct:

The Handle method updates the first 3, while the RunningState is reflecting the state machine state for the Observer. If we could do this, we could probably simplify the design.

Today the Observer grain is not partitioned, it sits as a coordinator that delegates to IObserverSubscriber implementations, which are partitioned. Any event going through the Observer grain goes through the Handle method that manages the state for the observer as a whole and then delegates the actual handling per partition to the subscriber. It is definitely a bottleneck and we can only go as fast as this is able to handle.

woksin commented 13 hours ago

Related to #1552