search

danielqsj / kafka_exporter

Kafka exporter for Prometheus
Apache License 2.0
2.1k stars 602 forks source link

Support MSK IAM SASL mechanism ? #380

Open ankush-custiv opened 1 year ago

ankush-custiv commented 1 year ago

The exporter currently cannot connect to AWS MSK with the IAM SASL mechanism, which is also the recommended way to connect to MSK. Using IAM mechanism, we will only require AWS_SECRET_ACCESS_KEY & AWS_ACCESS_KEY_ID to connect to MSK. Please let us know if it will be supported in future.

jclRatepay commented 1 year ago

I am also very interested in this feature as we would like to make use of the kafka_exporter to get the metrics off of our AWS MSK clusters. I had a first look into the required changes my self - and am afraid that I guess it won't be so easy to achieve (need to mention that I have only poor golang experience yet). The kafka_exporter is making use of "sarama" as a Kafka Client which does not support MSK IAM SASL mechanism yet. There exist issues for implementing this in the client (see e.g. https://github.com/Shopify/sarama/issues/1985) which also seemed to be worked on, but this will be some kind of prerequisite before being able to implement that support here, and going through the sarama-library issues they seem to face some problems there.....

ankush-custiv commented 1 year ago

@jclRatepay Right, to support this feature, kafka_exporter must move to "kafka-go"(https://github.com/segmentio/kafka-go) instead of "sarama", atleast that's what we did at our org to use IAM SASL mechanism. There are many other benefits to using kafka-go over sarama & other kafka client libs, which are mentioned on their README. However, I'd wait for @danielqsj's feedback before going in this direction.