Will this library be released for other languages, like .net or nodejs?

[robertbarron]

Hello,

We've been using a production level Amazon MSK for about a year now and saw the news about IAM integration. Our company has a few hundred microservices that connect to Amazon MSK via TLS. These microservices are build on .NET or Nodejs.

So, the main question is if you guys are planning on releasing this library for .NET, nodeJS or other languages.

Thanks

[azhurbilo]

plus "python" 🙂

[grsubramanian]

Hi thanks for your comment and interest in using IAM access control for MSK.

Many of the Kafka clients for non-JVM languages use a C-based library called librdkafka. This will be the place that this authentication mechanism needs to be supported. From an AWS perspective, it would be great to hear about any other specific language support you might need there.

[grsubramanian]

There now seems to be an ongoing discussion about this in the librdkafka community. See https://github.com/edenhill/librdkafka/discussions/3385.

[Nevon]

A discussion has also been opened regarding KafkaJS, which is not binding to librdkafka. See tulios/kafkajs#1098.

[andormarkus]

Hi All,

Our dataflow looks like this [DMS, Lambda(python)] -> MSK -> EMR Spark (python). I hope AWS add this feature to DMS and EMR Spark (python) soon.

On python there are two main kafka packages:

• confluentinc / confluent-kafka-python - is a python wrapper around librdkafka
• dpkp / kafka-python - is a pure python implementation, that follows Java client's implementation as close as possible. kafka-python in not binding to librdkafka as well

On kafka-python I have started discussion. See https://github.com/dpkp/kafka-python/issues/2232

(update): We are using terraform (Mongey / terraform-provider-kafka) to configure our kafka cluster. Go support would be nice as well.

[twmb]

For potential cross-language implementation reference, I've added support for AWS_MSK_IAM in my franz-go client. I've tested it against a scrappy cluster setup, but I'd be happy to have broader testing (and I'm planning to turn down my cluster imminently).

[Nevon]

I would be really helpful if there was a specification somewhere explaining how this authentication mechanism is supposed to work, so that each library author doesn't have to reverse engineer the Java implementation. @twmb's Go implementation is at least easier to read, but it would be preferable to not have to rely on a game of telephone.

EDIT: @twmb could you share how you managed to work against the MSK cluster? I'm getting what looks like a nonsense response to the ApiVersions request and then the connection is immediately closed, so I don't even get to the authentication flow.

{"level":"DEBUG","timestamp":"2021-05-23T14:40:47.947Z","logger":"kafkajs","message":"[Connection] Connecting","broker":"b-1.kafkajs-test.y9i95e.c3.kafka.eu-north-1.amazonaws.com:9098","clientId":"kafkajs","ssl":false,"sasl":true}
{"level":"DEBUG","timestamp":"2021-05-23T14:40:47.954Z","logger":"kafkajs","message":"[Connection] Request ApiVersions(key: 18, version: 2)","broker":"b-1.kafkajs-test.y9i95e.c3.kafka.eu-north-1.amazonaws.com:9098","clientId":"kafkajs","correlationId":1,"expectResponse":true,"size":21}

# Me logging what I get on the socket
<Buffer 15 03 03 00 02 02 50>

{"level":"DEBUG","timestamp":"2021-05-23T14:40:47.963Z","logger":"kafkajs","message":"[Connection] Kafka server has closed connection","broker":"b-1.kafkajs-test.y9i95e.c3.kafka.eu-north-1.amazonaws.com:9098","clientId":"kafkajs"}

[grsubramanian]

@Nevon it is necessary to use TLS encryption when working with IAM access control. Plaintext channels are not supported.

Meanwhile, we are in the process of improving the documentation of the SASL mechanism, so it becomes easier to implement in other languages.

[Nevon]

it is necessary to use TLS encryption when working with IAM access control. Plaintext channels are not supported.

Ah, thank you! 🤦‍♂️ Now I'm getting something that looks like it makes more sense.

[twmb]

@Nevon fwiw I usually use my kcl command line program to test things, and I added support for AWS_MSK_IAM to it. Let me know if you try it, I'd be pretty stoked :)

edit1: also, I recommend using the latest commit, which has a slight change in how the configuration is loaded edi2: also, I've pushed a few small updates to the AWS_MSK_IAM implementation to add support for session tokens and user agents, so it is worth it to check out the latest version for implementation reference.

[sayantacC]

I would be really helpful if there was a specification somewhere explaining how this authentication mechanism is supposed to work, so that each library author doesn't have to reverse engineer the Java implementation. @twmb's Go implementation is at least easier to read, but it would be preferable to not have to rely on a game of telephone.

We have updated the README to add details on how the authentication payload is generated. It would be really great if you could let us know if this helps.

[Nevon]

Thanks for adding the additional information to the readme. I was working on an initial implementation using @aws-sdk/signature-v4 yesterday, but my request was rejected, so I think I'll need to do some wire capture to see where I'm going wrong. Hopefully I'll have some time to go back to it this weekend.

[sayantacC]

@Nevon Please let us know if we can help by looking at some code, even if it is an early version.

[Nevon]

Haven't had any time to work on this yet, as some personal stuff had to take priority. If anyone else wants to take a crack at it and provide a third-party plugin, definitely don't let me stop you. There's no telling if I'll have the time to finish this.

I created https://github.com/tulios/kafkajs/pull/1101 to allow for injecting custom authentication mechanisms at runtime. I've been building the msk iam auth plugin using this interface. There are some things I'm not super happy with in terms of the typings, so maybe there will be some changes there to make it easier to work with, but overall that's how you'd extend KafkaJS with a new authentication mechanism.

[jmaver-plume]

I managed to connect to MSK with IAM with Node.js and kafkajs by not using @aws-sdk. Instead I manually wrote all the code required for generating authentication payload, signature, canonical request etc. The part where I am having trouble is with generating temporary credentials (access key, secret key, token). Currently I am using the following way to generate temporary credentials: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html#instance-metadata-security-credentials Does Node.js @aws-sdk package have a way to generate (and retrieve) temporary credentials?

TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"` \
&& curl -H "X-aws-ec2-metadata-token: $TOKEN" -v http://169.254.169.254/latest/meta-data/iam/security-credentials/s3access

[sayantacC]

The Node.js aws sdk supports credential providers which can extract temporary credentials. Please check if this will serve your purpose:

https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/modules/_aws_sdk_credential_provider_node.html

It describes how the credential provider can extract credentials from an instance role or container role or assume another role.

On another note, are you planning to contribute this integration to kafkajs?

Sayantan

On Jul 9, 2021, at 5:15 PM, Jaka Maver @.***> wrote:



I managed to connect to MSK with IAM with Node.js and kafkajs by not using @aws-sdk. Instead I manually wrote all the code required for generating authentication payload, signature, canonical request etc. The part where I am having trouble is with generating temporary credentials (access key, secret key, token). Currently I am using the following way to generate temporary credentials: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html#instance-metadata-security-credentials Does Node.js @aws-sdk package have a way to generate (and retrieve) temporary credentials?

TOKEN=curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600" \ && curl -H "X-aws-ec2-metadata-token: $TOKEN" -v http://169.254.169.254/latest/meta-data/iam/security-credentials/s3access

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/aws/aws-msk-iam-auth/issues/10#issuecomment-877518883, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AIA3ZZRXQPRD7TW3YRDYOHTTW6GHVANCNFSM44IFMBGA.

[jmaver-plume]

Thank you. I will check that out. Yes, I am planning on open sourcing the integration, but we will see if we can integrate it into core kafkajs or keep it as a separate authentication mechanism plugin.

[jmaver-plume]

@sayantacC Hey, can you explain to me what does EXPIRES_IN header mean? Does this mean that every let's say 900 seconds the connection needs to be reauthenticated or something else? If we set EXPIRES_IN to 10 seconds then we need to generate new signature every 10 seconds and if we set it to 1 week we need to generate it once every week?

[grsubramanian]

The x-amz-expires field represents how long the signature is valid, so the request made with signature computed at time T needs to be authenticated before T + x-amz-expires. However, once the request has been authenticated and connection to the broker authorized, the connection will last until one of the following happens.

1. 12 hours elapse.
2. The IAM assumed role on the client side expires (this expiration is upper-bounded by 12 hours).
3. The client closes the connection for any reason.
4. The broker crashes and closes the connection.

In cases 1 and 2 above, the client will need to reauthenticate in order to continue making any further Kafka API requests over the same connection. If the client reauthenticates successfully, then it can continue using the same connection, else the broker will terminate the connection.

[akshayar]

Hi , I have a customer who is using MSK and wants to use IAM for data plan Auth&Auth. They are using .Net library. What is the way forward for them?

[mwatson-x]

Hi thanks for your comment and interest in using IAM access control for MSK.

Many of the Kafka clients for non-JVM languages use a C-based library called librdkafka. This will be the place that this authentication mechanism needs to be supported. From an AWS perspective, it would be great to hear about any other specific language support you might need there.

Is this still true? The people at librdkafka say that this is something that they will not implement and is the responsibility of AWS. https://github.com/edenhill/librdkafka/issues/3780#issuecomment-1080451296

If it is, we then have a stalemate and really important functionality won't be implemented

[markl11]

Please create a Python package to support MSK Serverless (don't rely on third parties to do this when it seems they won't!)

[brainwipe]

We're a .NET Core shop that is using MSK with IAM and the Confluent Kafka library. Any news on a .NET Core plugin for IAM or is the solution to move the cluster to a different auth framework?

[simonkarman]

First of all: +1 on this issue, since I would also love to see a release of this library for C# and NodeJS. As there seems to be no progress on this for 1.5 years I do have another question related to it.

As long as there is no support for C# and/or NodeJS, we thought of setting up a Java Kafka proxy that uses the aws-msk-iam-auth library instead. The client application (C# or NodeJS) can then simply connect via this Kafka proxy (running on localhost) to connect to the MSK cluster with IAM. All we need for this is a Kafka proxy implementation that can use the aws-msk-iam-auth library. For this I looked at the following two options:

1) dajudge/kafkaproxy - A Kafka proxy written in Java using GraalVM. Unfortunately it doesn't seem to support the SASL mechanism at all and it seems like the aws-msk-iam-auth library can not be easily added to it.

2) grepplabs/kafka-proxy - A Kafka proxy written in Go. Although this supports SASL, it expects a path to an authentication plugin binary, which the aws-msk-iam-auth is not.

I don't see a clear way forward with this approach either. Is there another kafkaproxy someone knows of that does actually support the aws-msk-iam-auth? Or is there another way to approach this entirely?

[niemyjski]

+1 for .NET support

[van4oza]

kafka-python + SASL MSK worker for me

producer = KafkaProducer(
    bootstrap_servers=SASL_SERVERS,
    value_serializer=lambda v: json.dumps(v).encode('utf-8'),
    api_version=(2,8,1),
    security_protocol='SASL_SSL',
    sasl_mechanism='SCRAM-SHA-512',
    sasl_plain_username=os.getenv('KAFKA_USER'),
    sasl_plain_password=os.getenv('KAFKA_PASS'),
    retries=1
)

[aidanmelen]

Is there another kafkaproxy someone knows of that does actually support the aws-msk-iam-auth? Or is there another way to approach this entirely?

@simonkarman Here is an example of Confluent's Kafka Rest Proxy connecting with MSK using the SASL/IAM endpoint. I know the proxy will start up and connect with MSK; however, I have not extensively tested the proxy to ensure it is fully functioning.

[aidanmelen]

Hi thanks for your comment and interest in using IAM access control for MSK.

Many of the Kafka clients for non-JVM languages use a C-based library called librdkafka. This will be the place that this authentication mechanism needs to be supported. From an AWS perspective, it would be great to hear about any other specific language support you might need there.

@grsubramanian Are you aware of this thread in the librdkafka. It seems like AWS should step up and use its vast resources to contribute back to the open source project. Especially when you consider the open-source community would need to run potentially costly MSK servers to dev/test. Otherwise, I am afraid we are in a stalemate and nothing will get done.

[grsubramanian]

@sayantacC is currently the best POC on this.

[tkarthick81]

did anyone got a solution for .Net application ? @robertbarron

[mbuotidem]

@tkarthick81 https://github.com/aws/aws-msk-iam-sasl-signer-net#getting-started might help

[robertcoltheart]

I don't think this gives the full functionality. From here:

- Non-Java client configuration using SASL_OAUTHBEARER mechanism
- Java client configuration using SASL_OAUTHBEARER mechanism or AWS_MSK_IAM mechanism

[sankalpbhatia]

Hi All, We are happy to share that you can now use AWS IAM authN/Z for your MSK clusters using Amazon MSK's open-sourced client helper libraries for various languages. The new feature allows customers to send IAM sigv4 signatures via SASL/OAUTHBEARER, an open standard for authorization and authentication. For more details and getting started, check the README sections of the client libraries.

Python: https://github.com/aws/aws-msk-iam-sasl-signer-python Golang: https://github.com/aws/aws-msk-iam-sasl-signer-go JS: https://github.com/aws/aws-msk-iam-sasl-signer-js C#/.NET: https://github.com/aws/aws-msk-iam-sasl-signer-net

This feature is now available to all existing MSK Serverless and Provisioned clusters.

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

[NaodFessahaye]

Hi All, We are happy to share that you can now use AWS IAM authN/Z for your MSK clusters using Amazon MSK's open-sourced client helper libraries for various languages. The new feature allows customers to send IAM sigv4 signatures via SASL/OAUTHBEARER, an open standard for authorization and authentication. For more details and getting started, check the README sections of the client libraries.

Python: https://github.com/aws/aws-msk-iam-sasl-signer-python Golang: https://github.com/aws/aws-msk-iam-sasl-signer-go JS: https://github.com/aws/aws-msk-iam-sasl-signer-js C#/.NET: https://github.com/aws/aws-msk-iam-sasl-signer-net

This feature is now available to all existing MSK Serverless and Provisioned clusters.

Is this also implemented for C/C++?

[hongbo-miao]

Hi @NaodFessahaye we have same needs for C/C++, opened another feature request at https://github.com/aws/aws-msk-iam-auth/issues/186 Hopefully it can support soon!