out_kafka2: Broker pool to take care of fetching metadata

fluent / fluent-plugin-kafka

Kafka input and output plugin for Fluentd

Other

303 stars 176 forks source link

out_kafka2: Broker pool to take care of fetching metadata #503

Closed dipendra-singh closed 1 month ago

dipendra-singh commented 9 months ago

Which issue(s) this PR fixes: https://github.com/fluent/fluent-plugin-kafka/issues/502

What this PR does / why we need it: On enabling the get_kafka_client_log true, this can be figured out the leader of partitions is being queried multiple times: here this metadata if refreshed by broker pool (here). Adding this in producer would resolve the issue. additionally, this is a part of an optimisation done on ruby-kafka.

Test cases passing post the changes.

Docs Changes: NA

dipendra-singh commented 8 months ago

@ashie Requesting review here

github-actions[bot] commented 1 month ago

This PR has been automatically marked as stale because it has been open 90 days with no activity. Remove stale label or comment or this PR will be closed in 30 days

kenhys commented 1 month ago

LGTM.

kenhys commented 1 month ago

Thanks @dipendra-singh !