[feature] Add AWS MSK IAM SASL Support for Kafka Consumer

[IgorYoda]

I would like to propose adding support for AWS MSK (Managed Streaming for Apache Kafka) with IAM-based SASL authentication to Centrifugo. This could be achieved by integrating the aws-msk-iam-sasl-signer-go library into the Kafka engine of Centrifugo.

Motivation AWS MSK is widely used for Kafka services, and using IAM-based authentication provides a secure way to manage access without the need for managing individual credentials. Currently, Centrifugo does not directly support IAM SASL authentication with Kafka, making it difficult to integrate with AWS MSK clusters using the managed IAM authentication method.

Proposed Solution Integrate the aws-msk-iam-sasl-signer-go library to provide native support for AWS IAM-based SASL authentication within the Kafka engine. Add configuration options to enable this authentication mechanism, such as: sasl.mechanism: "AWS_MSK_IAM" IAM role or credentials configuration (similar to other AWS SDKs). This would allow Centrifugo users to connect to AWS MSK Kafka clusters securely using IAM-based authentication without additional workarounds or custom middleware.

References: aws-msk-iam-sasl-signer-go AWS MSK IAM Documentation: https://docs.aws.amazon.com/msk/latest/developerguide/iam-access-control.html This feature would greatly enhance the integration of Centrifugo with AWS MSK, providing seamless connectivity and enhanced security.

[Vahanerevan]

+1

[FZambia]

Hello @IgorYoda

Centrifugo does not have Kafka Engine, it has async Kafka consumer which is a different thing. Engine is responsible for PUB/SUB and history in channels. Async consumers are responsible for consuming some external queue and execute server API commands passed through that queue. This makes me worry you probably misunderstood terms. Could you describe your use case - is this a feature for Kafka async consumer?

[IgorYoda]

Hello @FZambia

I recognize that Centrifugo uses an async Kafka consumer and does not have a Kafka Engine. The request is aimed at enabling IAM-based SASL authentication for Kafka async consumers to allow secure integration with AWS MSK clusters. This would be achieved by incorporating the aws-msk-iam-sasl-signer-go library and exposing relevant configuration options for Centrifugo users.

[FZambia]

Hello, added in https://github.com/centrifugal/centrifugo/pull/912 - should be released during next week.

[IgorYoda]

@FZambia Do you have dev docker images for test before release ?

[FZambia]

Do you have dev docker images for test before release ?

Unfortunately no, you can generally try things locally with https://centrifugal.dev/docs/getting-started/installation#build-from-source

AWS MSK IAM SASL released in https://github.com/centrifugal/centrifugo/releases/tag/v5.4.7

Docs updated

Note, in case of "aws-msk-iam" Centrifugo uses sasl_user and sasl_password options as access key and secret key when configuring AWS auth.

Please let me know whether it works, because I did not have a chance to somehow try it with real AWS Kafka

[FZambia]

Hello, any updates guys? Were you able to try it?