Support for AWS MSK IAM authorization

[prydonius]

Hi! We are looking at using Strimzi to deploy Kafka Connect clusters using Kafka brokers deployed using Amazon MSK. We'd like to use MSK's IAM feature for access control within the Kafka cluster (https://docs.aws.amazon.com/msk/latest/developerguide/iam-access-control.html), and I was curious how we could enable this when deploying Kafka Connect clusters using Strimzi.

The MSK docs say Kafka clients need to be configured with the following properties:

ssl.truststore.location=<PATH_TO_TRUST_STORE_FILE>
security.protocol=SASL_SSL
sasl.mechanism=AWS_MSK_IAM
sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler

and the client requires the aws-msk-iam-auth plugin. This is fine because I can build an image for Strimzi containing this plugin.

The problem is, according to the Strimzi docs, there are specific ways to configure authn/authz and although the SASL_SSL protocol is supported, there doesn't seem to be a way to configure the sasl.mechanism and other options. I also cannot configure these manually in the KafkaConnect CR .spec.config because the docs suggest that any ssl., security. or sasl. are ignored in that block. Is there any other way we can configure these options with Strimzi?

[scholzj]

This is currently not supported. But judging from your description, it should not be too hard to add.

[Anthuang]

Hey, we're looking for the same functionality as @prydonius, and are considering to contribute support for it.

Curious what the work looks like to add support for this (and if possible could you point us in the right direction)?

Much appreciated!

[scholzj]

@Anthuang I think it might be best to have a look at the #5483 issue, where we discussed a possibility to have a custom authentication implemented => that should allow to use both the MSK IAM as well as other types of authentication. So having a look at that discussion might be the best start.

Technically, it would mean new authentication type which would as a field probably pass the whole SASL JAAS config in the options and anything else what might be needed.

[anandswaminathan]

We were able to get this work. We changed the https://github.com/strimzi/strimzi-kafka-operator/blob/main/docker-images/kafka-based/kafka/scripts/kafka_connect_config_generator.sh to support AWS IAM.

Also this blog post is super useful as we use Connect with Debezium - https://thedataguy.in/debezium-with-aws-msk-iam-authentication/

Once Custom Auth is available, we will look to Switch to that.

[fjbecerra]

Hey, If this is ready for development, I'd like to give it a go, @scholzj assign it to me please!

[scholzj]

This should follow the (in progress) proposal from @MaayanFarchi: https://github.com/strimzi/proposals/pull/42. The proposal https://github.com/strimzi/proposals/pull/41 is another example - but this is for the server side and not client side.

The idea is that there should be a custom authentication which allows you to:

• Add your own custom authenticator such as the MSK IAM one to the container images
• Configure it by more or less directly specifying the Kafka properties such as the JAAS configuration etc.

[scholzj]

Triaged on 7.6.2022: The proposal strimzi/proposals#42 seems stalled. But it is the right way to go for this issue. To have a custom authentication which allows users to add their own authenticators (be it for Amazon AWS or something else) and use it from the custom resources.

[ekeric13]

New to the strimzi space so just a general question: Is it the operator that needs to be able to authenticate with MSK IAM since it is talking to the brokers as a proxy for the strimzi CRDs? Or does the underlining image used for the KafkaConnect CRD need to be able to authenticate with MSK IAM since it talks directly to the brokers?

If it is the kafkaConnect container talking directly to the broker can't you just use your own custom docker image that supports MSK IAM? https://strimzi.io/docs/operators/latest/deploying.html#creating-new-image-from-base-str

[ekeric13]

More specific question for @anandswaminathan.. In order to get it working by changing this file what were the exact steps you took? It seems that all that needs to be done to that file is add sasl.client.callback.handler.class and expose a corresponding ENV variable. But where to go from there? Do you build this docker image and then use it as a base to the operator? to the kafka connect image?

[anandswaminathan]

I had to add a bunch of values. The list is available in this blog- https://thedataguy.in/debezium-with-aws-msk-iam-authentication/

Yes, after that build the image for the Kafka Connect, so that Connect can startup with the right properties.

[ekeric13]

Okay so the changes were propagated to the kafka connect image you used... NOT the strimzi operator. So I guess that means kafka connect still talks to the brokers, and not the strimzi operator?

And if that is the case can't authenticating with msk via IAM be solved by specifying additional properties in here: https://strimzi.io/docs/operators/latest/configuring.html#type-KafkaConnectSpec-reference if you are using a custom image that supports them like cp-kafka-connect. https://docs.confluent.io/platform/current/installation/configuration/connect/index.html#connectconfigs_sasl.client.callback.handler.class

Assuming the KafkaConnect also supports all of the ones in the kafka docs: https://kafka.apache.org/documentation/#connectconfigs_sasl.login.callback.handler.class

[scholzj]

This is about the clients such as Connect or Mirror Makers being able to talk to Amazon MSK using the IAM authentication. The operator it self does not talk to any Amazon MSK brokers. You can of course add the library to the custom container image. But you also need the API in the custom resources to be able to configure and use it which is what the proposal is really about. That way, you would be able to use it just by adding the JARs, but not modify any startup scripts or Java code.

[ekeric13]

Okay I am following now. If we were to hardcode these values:

security.protocol=SASL_SSL
sasl.mechanism=AWS_MSK_IAM
sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler

in a .properties file when building the image would that be a workaround? Specifically what I have in mind is the cp-kafka-connect image lets you set ENV variables that would update the .properties file:

https://docs.confluent.io/platform/current/installation/docker/config-reference.html#kconnect-long-configuration

[scholzj]

The Strimzi images are designed to be used through the operator and using the Strimzi CRD APIs. So no, doing this is not easy and you would need to change the either the shell scripts or the operator code.

[sherifkayad]

@scholzj are there any updates on this issue? We are also interested in using the standalone Topic operator from Strimzi to manage our MSK Kafka Topics .. our clusters are using the MSK IAM Auth ..

[scholzj]

This issue covers the operands such as Kafka Connect or Mirror Maker. It does not cover the Topic Operator in any way. I do not know if you can configure Topic Operator to use Amazon IAM.

[sherifkayad]

@scholzj thanks for your prompt reply .. what can I do to verify this? .. The documentation only mentions the following: If you want to connect to Kafka brokers that are using certificates from a public certificate authority, set STRIMZI_PUBLIC_CA to true. Set this property to true, for example, if you are using Amazon AWS MSK service. (https://strimzi.io/docs/operators/latest/deploying.html#deploying-the-topic-operator-standalone-str)

However, nothing is mentioned about the MSK IAM Auth ..

[scholzj]

As it is unrelated, maybe you can start some discussion thread about this topic to avoid unnecessary discussions on this issue?

[TFiroozian]

@scholzj wondering if there are any updates on this issue?

[scholzj]

You mean the IAM authentication? No, not yet I'm afraid.

[aidan-melen]

We are working on modifying the kafka_connect_config_generator.sh script to support KAFKA_CONNECT_SASL_MECHANISM="aws". It would integrate with the AWS environments variables and fit nicely into the spec.authentication.mechanism CRD.

To have full support we would need to add the add the uber jar to the classplath in the images. Otherwise, at the very least, it would be great to allow others to extend the base image to manage the aws-msk-iam-auth themselves. Right now, the kafka_connect_config_generator.sh will overwrite any custom jaas or handler configOverrides.

@scholzj would you be open to accepting these changes? if so, would you consider full support (packaging the jar) or partial support by allowing others to extend the base image?

[scholzj]

This was discussed multiple times ... we cannot add any specific support for AWS authentication without:

• Some reasonable guarantees that AWS will keep it up to date with reasonable timelines
• Some test environment to test it

Unless you can provide those, the right way would be to add support for some type: custom authentication which allows you to customize the options currently set by the operator such as the sasl.mechanism or sasl.jaas.config. That way, you can add the AWS JAR to the container image and use this authentication type to set it up. But at the same time, we do not need AWS for testing it since it is just a generic custom authentication solution -> so the only thing to test in Strimzi is that the values are passed properly.

If you are interested in contributing that, a proposal would be the right way to start: https://github.com/strimzi/proposals

PS: I do not understand what you cannot extend in the container image. You can extend it any way you want as long as you keep the original Strimzi structure and interface scripts.

[aidan-melen]

We can take a look at a formal purposal

[aidan-melen]

PS: I do not understand what you cannot extend in the container image. You can extend it any way you want as long as you keep the original Strimzi structure and interface scripts.

Imagine I have placed the aws-iam-msk-auth uber jar in the path on an extended strimzi image and now I am tasked with configuring the general configuration:

# Sets up TLS for encryption and SASL for authN.
security.protocol = SASL_SSL

# Identifies the SASL mechanism to use.
sasl.mechanism = AWS_MSK_IAM

# Binds SASL client implementation.
sasl.jaas.config = software.amazon.msk.auth.iam.IAMLoginModule required;

# Encapsulates constructing a SigV4 signature based on extracted credentials.
# The SASL client bound by "sasl.jaas.config" invokes this class.
sasl.client.callback.handler.class = software.amazon.msk.auth.iam.IAMClientCallbackHandler

We cannot use the spec.authenitcation.mechanism because it is very opinionated in the current form. Therefore we have two alternatives to consider: configOverrides and pod environment variables.

• configOverrides will not work because the generate script will override the security.protocol among others.
• the environment variable don't work out of the box because there is no mechanism to interpenetrate the values.

[scholzj]

I don't follow the issue. If you want to use the image with Strimzi, you can add the AWS JAR. But you have to respect what the operator does. An operator will be always opinionated. It needs to understand what it is doing. If you want to configure everything in your cluster by hand, an operator might not be the right choice for you.

[aidanmelen]

PS: I do not understand what you cannot extend in the container image. You can extend it any way you want as long as you keep the original Strimzi structure and interface scripts.

I opened a proposal going more into depth how image extension could improved to allow for more custom configuration without burdening the Strimzi project with support many custom cloud provider authetniction jars and plumbing the customer configuration properties.

[scholzj]

I think you are missing the point of an operator. If you want to write your Connect options into an environment variable and have that used in some deployment, do not use an operator, use a Helm Chart or write the Deployment yourself.

[aidanmelen]

I would prefer to use the strimzi operator as it has a number of benefits over a simple helm chart. Perhaps I will just extend the kafka-base image the custom config generator scripts while I am adding the custom authentication JAR. That way I can configure bootstrap.servers, group.id, internal topics, etc using the CRD. I will close my other proposal.

Update - I created an example project for strimzi Kafka connect using SASL/IAM using image extension.

[mickmorse]

Did the AWS MSK IAM Authentication option for KafkaConnect get merged?

[aidanmelen]

Did the AWS MSK IAM Authentication option for KafkaConnect get merged?

not officially. but you can see the example project for strimzi Kafka connect using SASL/IAM using image extension as inspiration.

[davido912]

This is something we would also love seeing.

[vl-kp]

how to manage MSK topics with strimzi via IAM auth?

[john-mcpeek]

I want to add a custom sasl.mechanism to the topic operator, but I haven't been able to get the code to build reliably. I followed the steps in the Dev Guide. Where can I get some help? The change to the Topic Operator config seems simple, but if I can't get the tests to run reliably, I can't know if it works.

Thanks.

This is my system: Windows 11 Ubuntu 22.04.3 LTS (WSL2) ZSH (also tried bash) Java version: 17.0.9 make - GNU Make 4.3 Apache Maven 3.8.4 (9b656c72d54e5bacbed989b64718c159fe39b537) Asciidoctor 2.0.16 Asciidoctor PDF 2.3.10 using Asciidoctor 2.0.16 yq (https://github.com/mikefarah/yq/) version v4.40.5 Docker - 24.0.6 ShellCheck - version: 0.8.0 Kubernetes - v1.28.2 (Docker desktop)

[scholzj]

@john-mcpeek Maybe best to start a separate discussion (https://github.com/orgs/strimzi/discussions) and share what from the build does not work for you. Maybe something will ring a bell. That said, I don't think anyone from the Strimzi core team uses Windows.

[debu99]

When MSK IAM auth can be supported?

[dlydiard]

Update - I created an example project for strimzi Kafka connect using SASL/IAM using image extension.

this example seems to only work if both the source and target are using IAM/SASL_SSL. Do you have any pointers on getting mixed mode working, where the source is SSL and the target is IAM?

[kapishreshth]

@scholzj Is there any official update on this MSK-IAM support?

[scholzj]

@kapishreshth No, there is no update. Nobody wrote a proposal to start with this.