search

knative-extensions / eventing-kafka-broker

Alternate Kafka Broker implementation.
Apache License 2.0
170 stars 117 forks source link

Support AWS MSK's IAM authentication via SASL_OAUTHBEARER #3514

Closed brianV closed 2 months ago

brianV commented 10 months ago

Problem AWS MSK is AWS's managed Kafka service. It offers an IAM access mode that restricts access based on IAM policies. They have a Serverless option, which only supports IAM access control.

You can authenticate this way by using the SASL_OAUTHBEARER mechanism as described here: https://docs.aws.amazon.com/msk/latest/developerguide/iam-access-control.html#configure-clients-for-iam-access-control

The underlying Signer library for Go generates an OAuth token from AWS credentials in the current AWS credential provider chain. The README shows an example using Sarama here: https://github.com/aws/aws-msk-iam-sasl-signer-go/tree/main

Persona: Which persona is this feature for? Event producers and consumers

Exit Criteria Users can specify configuration options for a Kafka Broker that will allow it to authenticate using SASL_OATHBEARER and a token provider that generates tokens from the user's default IAM credential provider chain.

Time Estimate (optional): How many developer-days do you think this may take to resolve?

Additional context (optional) Add any other context about the feature request here.

Cali0707 commented 10 months ago

cc @pierDipi @creydr

github-actions[bot] commented 7 months ago

This issue is stale because it has been open for 90 days with no activity. It will automatically close after 30 more days of inactivity. Reopen the issue with /reopen. Mark the issue as fresh by adding the comment /remove-lifecycle stale.

brianV commented 7 months ago

Regarding the above lifecycle bot comment - should feature requests be closed just because they haven't been prioritized?

pierDipi commented 7 months ago

@brianV thanks for the request, our concern is that we will not be able to test the integration with AWS MSK and the library is AWS specific.

Is there any way to implement the integration with a generic oauth library that would work with components like Strimzi + IDP (like keycloak) so that we can test and verify it works and doesn't break in future releases?

/remove-lifecycle stale

pierDipi commented 7 months ago

I'm not sure if the approach suggested here would work with AWS identities https://github.com/IBM/sarama/issues/1320#issuecomment-570231328

adrien-f commented 6 months ago

Greetings 👋

I am currently implementing MSK IAM authentication for Keda at https://github.com/kedacore/keda/pull/5692 and based on that experience I would be open also to contributing it to Knative.

I understand the worries about not being able to try out the integration with a real cluster. It is possible to implement SASL_OAUTBEARER similar to Keda:

Speaking of Keda, I know that the source can be scaled up and down with it, I suppose that once merged, this would also work with MSK brokers.

Does that sound alright?

github-actions[bot] commented 3 months ago

This issue is stale because it has been open for 90 days with no activity. It will automatically close after 30 more days of inactivity. Reopen the issue with /reopen. Mark the issue as fresh by adding the comment /remove-lifecycle stale.