search

fluent / fluent-plugin-kafka

Kafka input and output plugin for Fluentd
Other
303 stars 178 forks source link

feat(out_kafka2): adds support for AWS IAM authentication to MSK usin… #481

Closed raytung closed 1 year ago

raytung commented 1 year ago

…g long lived credentials

Addresses https://github.com/fluent/fluent-plugin-kafka/issues/472

This PR adds support for AWS IAM authentication using long lived credentials (access key id and secret access keys). To support AWS assume role and STS, we will need to wait for upstream's ruby-kafka library support.

We will need to bump ruby-kafka to 1.5.0 in order to support this feature.

ashie commented 1 year ago

It's not this PR's problem but I think it would be better to unify indent & setting mandatory parameters.

ashie commented 1 year ago

we will need to wait for upstream's ruby-kafka library support.

BTW according to https://github.com/zendesk/ruby-kafka ruby-kafka is no longer maintained. We might need to consider deprecating also this plugin...

raytung commented 1 year ago

@ashie I saw that too 😢 I spent a bit of time digging up the story for librdkafka but it looks like there's ~a heavy~ (perhaps overlooked) push back from the author / Confluent to support pluggable authentication call back in librdkafka which means there's no way to support AWS IAM auth 😢 (Basically they are not implementing this KIP https://cwiki.apache.org/confluence/display/KAFKA/KIP-86%3A+Configurable+SASL+callback+handlers)