search

moleculerjs / moleculer-channels

Reliable messages for Moleculer services via external queue/channel/topic.
MIT License
75 stars 15 forks source link

Feature Request: Batching by Time Window or Event Count #46

Open valeeum opened 2 years ago

valeeum commented 2 years ago

Would it be feasible to add a feature that allows to buffer/queue incoming events for a period of time and/or a maximum event count before firing the channel handler? There's a similar feature in AWS lambda and it would be very useful in building out data pipelines where bulk data operations can more efficiently insert/upsert data. If i'm overthinking this and there's an easier workaround or solution, please let me know.

https://aws.amazon.com/about-aws/whats-new/2020/11/aws-lambda-now-supports-batch-windows-of-up-to-5-minutes-for-functions/

ujwal-setlur commented 1 year ago

This would be very useful indeed!

intech commented 1 year ago

You can solve this with Benthos and add it to every node in the pipeline. There is Message Batching.