zalando / nakadi

A distributed event bus that implements a RESTful API abstraction on top of Kafka-like queues
https://nakadi.io
MIT License
952 stars 292 forks source link

reassign topics in multitimeline consumer #1516

Closed adyach closed 1 year ago

adyach commented 1 year ago

One-line summary

reassign topics in multitimeline consumer

the commit is a part of the bigger work to create multi-timeline consumer pool for subscriptions statistics. that's needed because in order to get statistics, the consumer has to be created for every partition for every request and read actual events. with the bigger load the amount of consumers created per second can reach very big numbers, which affects Kafka performance negatively and creates issues on another endpoints for example increased publishing latency. in order to avoid that without changing the architecture of the statistics endpoint, the pool of consumers can be created and consumers can be safely reused. and here comes the current commit, which enables us to reuse the multi-timeline consumer utilising reassign method otherwise consumer is thrown away after every assigning of new topic partition.

a1exsh commented 1 year ago

:+1:

adyach commented 1 year ago

👍