search

logstash-plugins / logstash-input-kafka

Kafka input for Logstash
Apache License 2.0
139 stars 122 forks source link

Make kafka client unique across multiple pipelines #305

Closed w32-blaster closed 4 months ago

w32-blaster commented 5 years ago

The current implementation makes the client_id unique when using multiple threads, but when using one more more threads with multiple pipeline configurations, we are getting the error javax.management.InstanceAlreadyExistsException: kafka.consumer:type=app-info,id=logstash-0 This is because every pipeline is trying to use the same client_id to connect to kafka. A workaround is to explicitly specify the client_id using a unique string.

The patch bellow is adding an additional suffix to the client_id after the thread # which is the pipeline_id. With this, we can be sure that we don't use the same client_id in a multi pipeline setup.

w32-blaster commented 5 years ago

is there any reason why this PR isn't reviewed at all by the maintainers?

robbavey commented 5 years ago

@w32-blaster The integration tests are failing because the way that the tests are currently written, no execution_context is supplied. If you want me to fix up the tests in this commit, let me know.

w32-blaster commented 5 years ago

Please do it. I don’t have enough context to fix that. Thanks

w32-blaster commented 5 years ago

@robbavey, can you help me to fix this please?

zhaoxinjie commented 5 years ago

is it necissry that client_id unique across multiple piplines? our logstash cluter distribute on 3 server and use the same client, it seems nothing wrong.