Open nicoodle opened 2 years ago
Had a little play - Created the following Dockerfile
FROM devshawn/kafka-gitops
RUN cd /usr/local/openjdk-8/lib && \
curl -LJO https://github.com/aws/aws-msk-iam-auth/releases/download/v1.1.1/aws-msk-iam-auth-1.1.1-all.jar -o /usr/local/openjdk-8/lib/.
MSK IAM Jar now exists in the docker image. Used environment variables which look like this:
KAFKA_BOOTSTRAP_SERVERS=REDACTED:9098
KAFKA_SECURITY_PROTOCOL=SASL_SSL
KAFKA_SASL_MECHANISM=AWS_MSK_IAM
KAFKA_SASL_JAAS_CONFIG=software.amazon.msk.auth.iam.IAMLoginModule required;
KAFKA_SASL_CLIENT_CALLBACK_HANDLER_CLASS=software.amazon.msk.auth.iam.IAMClientCallbackHandler
Verbose output
Generating execution plan...
11:11:26.016 [main] INFO com.devshawn.kafka.gitops.config.KafkaGitopsConfigLoader - Kafka Config: {security.protocol=SASL_SSL, sasl.mechanism=AWS_MSK_IAM, sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;, bootstrap.servers=REDACTED:9098, sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler, client.id=kafka-gitops}
11:11:26.023 [main] INFO com.devshawn.kafka.gitops.service.ConfluentCloudService - Using ccloud executable at: ccloud
11:11:26.025 [main] INFO com.devshawn.kafka.gitops.service.ParserService - Parsing desired state file...
java.lang.NullPointerException
at com.devshawn.kafka.gitops.service.KafkaService.buildAdminClient(KafkaService.java:115)
at com.devshawn.kafka.gitops.service.KafkaService.getAcls(KafkaService.java:41)
at com.devshawn.kafka.gitops.manager.PlanManager.planAcls(PlanManager.java:137)
at com.devshawn.kafka.gitops.StateManager.generatePlan(StateManager.java:90)
at com.devshawn.kafka.gitops.StateManager.plan(StateManager.java:80)
at com.devshawn.kafka.gitops.cli.PlanCommand.call(PlanCommand.java:38)
at com.devshawn.kafka.gitops.cli.PlanCommand.call(PlanCommand.java:19)
at picocli.CommandLine.executeUserObject(CommandLine.java:1783)
at picocli.CommandLine.access$900(CommandLine.java:145)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2141)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2108)
at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:1975)
at picocli.CommandLine.execute(CommandLine.java:1904)
at com.devshawn.kafka.gitops.MainCommand.main(MainCommand.java:76)
I've even tried setting the CLASSPATH to /usr/local/openjdk-8/lib/aws-msk-iam-auth-1.1.1-all.jar with the same output.
It's possible I've missed something but it appears kafka-gitops is not compatible with MSK IAMs authentication.
I have also tested this with the same result as @nicoodle. I was however able (minimal testing so far) to enable SASL/SCRAM (in addition to IAM which we'll use for applications) and was able to get kafka-gitops working with the following environment variables after creating a Secret and associating it with the MSK Cluster:
KAFKA_BOOTSTRAP_SERVERS=XXXXXX.amazonaws.com:9096
KAFKA_SASL_JAAS_PASSWORD=XXXXXX
KAFKA_SASL_JAAS_USERNAME=XXXXXX
KAFKA_SASL_MECHANISM=SCRAM-SHA-512
KAFKA_SECURITY_PROTOCOL=SASL_SSL
Maybe not the solution you were looking for, but a work around until a better solution is available.
Hey!
We've been using kafka-gitops
with MSK and IAM auth like this:
$ export KAFKA_SASL_CLIENT_CALLBACK_HANDLER_CLASS="software.amazon.msk.auth.iam.IAMClientCallbackHandler"
$ export KAFKA_SASL_JAAS_CONFIG="software.amazon.msk.auth.iam.IAMLoginModule required;"
$ export KAFKA_SASL_MECHANISM="AWS_MSK_IAM"
$ export KAFKA_SECURITY_PROTOCOL="SASL_SSL"
$ java -cp ${AWS_MSK_IAM_PLUGIN_PATH}:${KAFKA_GITOPS_PATH} com.devshawn.kafka.gitops.MainCommand ...
With AWS_MSK_IAM_PLUGIN_PATH
being the path to the MSK IAM Auth plugin jar, and KAFKA_GITOPS_PATH
being the path to the kafka-gitops executable.
That works really well, as long as you have valid AWS credentials :)
Hi there!
This is such a fantastic project and it's going to be super useful for our usecase. I was just wondering if the standard docker container has MSK IAM authentication support?
Looking at the AWS documentation, you can see an extra class is required with a few extra configuration options. Is this currently supported by kafka-gitops? If not would it be as simple as placing the MSK class in the classpath within the container and setting the required properties?
Required properties:
Thanks in advance!