IBM / sarama

Sarama is a Go library for Apache Kafka.
MIT License
11.51k stars 1.75k forks source link

Telegraf - How does Sarama select which broker to connect to when a list is supplied #1872

Closed clever-trevor closed 1 year ago

clever-trevor commented 3 years ago

Question

The Telegraf monitoring agent has an output plugin which posts metrics to Kafka, utilising the Sarama project. https://github.com/influxdata/telegraf/tree/master/plugins/outputs/kafka

Within the Telegraf config, you specify a variable containing a string of one or more Brokers as follows brokers = ["localhost:9092","10.1.2.3:9092","1.2.3.4:9092"]

What I'm not clear on is how the selection is made on which broker to connect to, and the behaviour when that broker becomes unavailable.

Additional Info I took a quick look at the code and within the Telegraf kafka plugin, I believe this is the line which creates the Kafka connection producerFunc: sarama.NewSyncProducer,

(Forgive me if this is wrong, I'm not a Go programmer)

I initially raised this with InfluxData who support Telegraf (link below) in case that adds any additional info but they were unable to help as the Sarama code is not written by them https://community.influxdata.com/t/how-does-telegraf-kafka-output-determine-what-broker-to-use-when-multiple-are-speecified/17342

clever-trevor commented 3 years ago

Any update here?

AnuragJ05 commented 2 years ago

It looks like sarama picks the first one from the list and ignores the rest, Is this a bug?

github-actions[bot] commented 1 year ago

Thank you for taking the time to raise this issue. However, it has not had any activity on it in the past 90 days and will be closed in 30 days if no updates occur. Please check if the main branch has already resolved the issue since it was raised. If you believe the issue is still valid and you would like input from the maintainers then please comment to ask for it to be reviewed.

dnwe commented 1 year ago

That’s not the case. Sarama will shuffle those bootstrap addresses on request. It will cycle through them to gain connectivity and then uses the advertised listeners for each broker when determining how to connect to those