getsentry / arroyo

A library to build streaming applications that consume from and produce to Kafka.
https://getsentry.github.io/arroyo/
Apache License 2.0
39 stars 6 forks source link

fix(processor): Create strategy lazily on first message #317

Open untitaker opened 6 months ago

untitaker commented 6 months ago

When a series of rebalance events occurs, this avoids recreating the strategy from within callbacks entirely (making on_revoke in particular faster). Instead the strategy is created whenever the first message is being processed.

There were ideas to create the strategy eagerly before even subscribing to a topic. However, currently create_with_partitions requires knowledge of the currently assigned partitions. We would have to deprecate this argument if we wanted to make this happen.