unable to connect to aws-msk using library

[ahsan-n]

error getting :

Exception in thread "main" org.apache.kafka.common.config.ConfigException: Invalid value software.amazon.msk.auth.iam.IAMClientCallbackHandler for configuration sasl.client.callback.handler.class: Class software.amazon.msk.auth.iam.IAMClientCallbackHandler could not be found.
    at org.apache.kafka.common.config.ConfigDef.parseType(ConfigDef.java:728)
    at org.apache.kafka.common.config.ConfigDef.parseValue(ConfigDef.java:474)
    at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:467)
    at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:108)
    at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:142)
    at org.apache.kafka.clients.admin.AdminClientConfig.<init>(AdminClientConfig.java:216)
    at org.apache.kafka.clients.admin.Admin.create(Admin.java:61)
    at kafka.admin.TopicCommand$AdminClientTopicService$.createAdminClient(TopicCommand.scala:212)
    at kafka.admin.TopicCommand$AdminClientTopicService$.apply(TopicCommand.scala:216)
    at kafka.admin.TopicCommand$.main(TopicCommand.scala:57)
    at kafka.admin.TopicCommand.main(TopicCommand.scala)

how i loaded classpath:

export CLASSPATH=/home/ec2-user/aws-msk-iam-auth-1.1.3-all.jar

logged in as user root. Using instance profile with kafka full access.

client.properties:

# Sets up TLS for encryption and SASL for authN.
security.protocol = SASL_SSL

# Identifies the SASL mechanism to use.
sasl.mechanism = AWS_MSK_IAM

# Binds SASL client implementation.
sasl.jaas.config = software.amazon.msk.auth.iam.IAMLoginModule required;

# Encapsulates constructing a SigV4 signature based on extracted credentials.
# The SASL client bound by "sasl.jaas.config" invokes this class.
sasl.client.callback.handler.class = software.amazon.msk.auth.iam.IAMClientCallbackHandler

executing from dir:

/home/ec2-user/kafka_2.13-2.5.0/bin

home/ec-user:

rw-r--r-- 1 root root      669 Apr 24 08:15 aws-msk-iam-auth-1.1.3-all.jar
drwxr-xr-x 6 root root       89 Apr  8  2020 kafka_2.13-2.5.0
-rw-r--r-- 1 root root 61459093 Jul  6  2020 kafka_2.13-2.5.0.tgz

java version:

[root@ip-192-168-130-35 bin]# java --version
openjdk 11.0.14.1 2022-02-08 LTS
OpenJDK Runtime Environment Corretto-11.0.14.10.1 (build 11.0.14.1+10-LTS)
OpenJDK 64-Bit Server VM Corretto-11.0.14.10.1 (build 11.0.14.1+10-LTS, mixed mode)

command:

 ./kafka-topics.sh --bootstrap-server b-1.qa-msk.grzo233b.c151223.kafka.ap-southeast-1.amazonaws.com:9098 --create --replication-factor 2 --partitions 3 --topic my-topic --command-config client.properties

not sure what I am mistaking here.

[OneCricketeer]

The files in /home/ec-user should not be owned by the root user

chown -R ec2-user /home/ec2-user then logout from root account and try again

[m1mohamad]

facing the same issue on on macOS it was working then stopped suddenly with no changes ./kafka-topics.sh --bootstrap-server b-1.stage.kafka.eu-west-1.amazonaws.com:9096 --command-config ~/kafka/config/config.properties --list
Exception in thread "main" org.apache.kafka.common.config.ConfigException: Invalid value software.amazon.msk.auth.iam.IAMClientCallbackHandler for configuration sasl.client.callback.handler.class: Class software.amazon.msk.auth.iam.IAMClientCallbackHandler could not be found. at org.apache.kafka.common.config.ConfigDef.parseType(ConfigDef.java:744) at org.apache.kafka.common.config.ConfigDef.parseValue(ConfigDef.java:490) at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:483) at org.apache.kafka.common.config.AbstractConfig.(AbstractConfig.java:113) at org.apache.kafka.common.config.AbstractConfig.(AbstractConfig.java:146) at org.apache.kafka.clients.admin.AdminClientConfig.(AdminClientConfig.java:231) at org.apache.kafka.clients.admin.Admin.create(Admin.java:133) at kafka.admin.TopicCommand$TopicService$.createAdminClient(TopicCommand.scala:205) at kafka.admin.TopicCommand$TopicService$.apply(TopicCommand.scala:209) at kafka.admin.TopicCommand$.main(TopicCommand.scala:50) at kafka.admin.TopicCommand.main(TopicCommand.scala)

[RajasGujarathi]

Facing the same issue on a Amazon Linux EC2

[MichalMoravik]

Had a similar issue. The aws-msk-iam-auth-1.1.3-all.jar should be in the Kafka directory. In my case: ./kafka_2.12-2.8.1.

From your example, you placed it in the root:

rw-r--r-- 1 root root 669 Apr 24 08:15 aws-msk-iam-auth-1.1.3-all.jar drwxr-xr-x 6 root root 89 Apr 8 2020 kafka_2.13-2.5.0 -rw-r--r-- 1 root root 61459093 Jul 6 2020 kafka_2.13-2.5.0.tgz

---

In order to get the file and place it inside this folder, I simply ran the following command in kafka/libs folder: wget https://github.com/aws/aws-msk-iam-auth/releases/download/v1.1.1/aws-msk-iam-auth-1.1.1-all.jar

Make sure you're in kafka/libs when executing this command.

You should now have aws-msk-iam-auth-1.1.3-all.jar in your Kafka folder.

Find more here, step number 7: https://docs.aws.amazon.com/msk/latest/developerguide/create-topic.html

[shayneoneill]

Anyone work out where to put this thing on a macos homebrew install?

[jeffreygolden]

@shayneoneill I got this working by adding the file to this homebrew path: /opt/homebrew/Cellar/kafka/{$KAFKA_VERSION}/libexec/libs/

[vrmare]

@jeffreygolden I'm trying to do the same on my MAC but dint have success yet. Here's what I have:

current working dir:

/opt/homebrew/Cellar/kafka/kafka_2.13-3.6.0

❯ java -version

openjdk version "21.0.1" 2023-10-17
OpenJDK Runtime Environment Homebrew (build 21.0.1)
OpenJDK 64-Bit Server VM Homebrew (build 21.0.1, mixed mode, sharing)

I have the jar file in this dir:

/opt/homebrew/Cellar/kafka/kafka_2.13-3.6.0
❯ ls libs/ | grep aws
aws-msk-iam-auth-1.1.1-all.jar

my client.properties

security.protocol=SASL_SSL
sasl.mechanism=AWS_MSK_IAM
sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler

Trying to connect but still same issue:

./bin/kafka-topics.sh --command-config ./config/client.properties --bootstrap-server $server --list

What could I be missing here?

[Devarsh23]

We are facing the same issue when we are trying to use the IAM authentication from kafka connect. We are getting Invalid value software.amazon.msk.auth.iam.IAMClientCallbackHandler for configuration sasl.client.callback.handler.class: Class software.amazon.msk.auth.iam.IAMClientCallbackHandler could not be found.

We are using the confluentinc/cp-kafka-connect-base image with version 7.5.0. So, while creating our image, which will use the IAM auth, we are using the latest jar of version 2.0.3. In this image, we are copying this jar into the following path

1. /usr/share/java/
2. /etc-kafka-connect/jars
3. /usr/share/java/cp-base-new

We are running our kafka-connect in the distributed mode. We have also set the CLASSPATH in the docker file as an environment variable to this path /usr/share/java/. Also, we have set the plugin path for our kafka-connect to /usr/share/java,/usr/share/confluent-hub-components But after all this, we still get the same error mentioned above. Do you guys have any idea on how to resolve this issue?

[nieldejonghe]

@Devarsh23 we are facing the same issue, have you found a solution for this?