search

logstash-plugins / logstash-input-kafka

Kafka input for Logstash
Apache License 2.0
139 stars 122 forks source link

don't join group until pipeline is initialized to prevent premature rebalancing #314

Closed bczifra closed 4 years ago

bczifra commented 5 years ago

When a Kafka input is used it cannot begin processing until the rest of the pipeline has been initialized. However, its own initialization can trigger rebalancing within Kafka. With a large complex pipeline that may take several minutes to initialize, that Kafka input can be blocked for quite a while resulting in a delay of message processing from the assigned partition. In such cases, it would be useful for the Kafka input to wait to join the group until the pipeline is actually read to process events.

yaauie commented 4 years ago

This issue is mitigated indirectly with a separate change that went into Logstash 7.5.2 and 7.6.0, in which the inputs to a pipeline are not started until the worker-loop is fully initialized. See https://github.com/elastic/logstash/pull/11492