search

centrifugal / centrifugo

Scalable real-time messaging server in a language-agnostic way. Self-hosted alternative to Pubnub, Pusher, Ably. Set up once and forever.
https://centrifugal.dev
Apache License 2.0
8.39k stars 594 forks source link

[feature]  Add AWS MSK IAM SASL Support for Kafka Consumer #881

Open IgorYoda opened 1 month ago

IgorYoda commented 1 month ago

I would like to propose adding support for AWS MSK (Managed Streaming for Apache Kafka) with IAM-based SASL authentication to Centrifugo. This could be achieved by integrating the aws-msk-iam-sasl-signer-go library into the Kafka engine of Centrifugo.

Motivation AWS MSK is widely used for Kafka services, and using IAM-based authentication provides a secure way to manage access without the need for managing individual credentials. Currently, Centrifugo does not directly support IAM SASL authentication with Kafka, making it difficult to integrate with AWS MSK clusters using the managed IAM authentication method.

Proposed Solution Integrate the aws-msk-iam-sasl-signer-go library to provide native support for AWS IAM-based SASL authentication within the Kafka engine. Add configuration options to enable this authentication mechanism, such as: sasl.mechanism: "AWS_MSK_IAM" IAM role or credentials configuration (similar to other AWS SDKs). This would allow Centrifugo users to connect to AWS MSK Kafka clusters securely using IAM-based authentication without additional workarounds or custom middleware.

References: aws-msk-iam-sasl-signer-go AWS MSK IAM Documentation: https://docs.aws.amazon.com/msk/latest/developerguide/iam-access-control.html This feature would greatly enhance the integration of Centrifugo with AWS MSK, providing seamless connectivity and enhanced security.

Vahanerevan commented 1 month ago

+1

FZambia commented 1 month ago

Hello @IgorYoda

Centrifugo does not have Kafka Engine, it has async Kafka consumer which is a different thing. Engine is responsible for PUB/SUB and history in channels. Async consumers are responsible for consuming some external queue and execute server API commands passed through that queue. This makes me worry you probably misunderstood terms. Could you describe your use case - is this a feature for Kafka async consumer?

IgorYoda commented 1 month ago

Hello @FZambia

I recognize that Centrifugo uses an async Kafka consumer and does not have a Kafka Engine. The request is aimed at enabling IAM-based SASL authentication for Kafka async consumers to allow secure integration with AWS MSK clusters. This would be achieved by incorporating the aws-msk-iam-sasl-signer-go library and exposing relevant configuration options for Centrifugo users.

FZambia commented 2 days ago

Hello, added in https://github.com/centrifugal/centrifugo/pull/912 - should be released during next week.

IgorYoda commented 1 day ago

@FZambia Do you have dev docker images for test before release ?