search

akka / akka-projection

Akka Projections is intended for building systems with the CQRS pattern, and facilitate in event-based service-to-service communication.
https://doc.akka.io/docs/akka-projection/current/
Other
100 stars 34 forks source link

Support for grouped flows #593

Open johanandren opened 2 years ago

johanandren commented 2 years ago

The batching support is super nice, and as far as I understand it you can also achieve that with flows, as long as you emit a Done for every element downstream.

Batching the offset update in that case is for every element in the batch, or if you use .withSaveOffset(afterEnvelopes = 100, afterDuration = 500.millis) possible to align the batch sizes when the rate is high, however matching the afterDuration with the duration in a groupedWithin seems sketchy.

Would be nice with a batching signature, something like .groupedWithinFlow(Flow[Seq[Envelope], Done, _]) where the committing is done based on the batching of the envelopes into the flow or accepting a Flow[Envelope, Seq[Envelope], _] where the user can batch anyway they want and get offsets commited based on those batches.

patriknw commented 2 years ago

I like the idea, but not sure it's very important. I don't think there is a strong requirement to "emit a Done for every element downstream". I think it's ok to group them together and emit one for the group if you are ok with the caveats explained in https://doc.akka.io/docs/akka-projection/current/flow.html

johanandren commented 2 years ago

Ah, so just emitting the last ProjectionContext together with the Done that goes downstream would do the trick?

patriknw commented 2 years ago

I think so ™️