IAMClientCallbackHandler could not be found

[tzookb]

keep getting this error:

[2023-01-06 14:13:05,494] ERROR Unknown error when running consumer:  (kafka.tools.ConsoleConsumer$:43)
org.apache.kafka.common.config.ConfigException: Invalid value software.amazon.msk.auth.iam.IAMClientCallbackHandler for configuration sasl.client.callback.handler.class: Class software.amazon.msk.auth.iam.IAMClientCallbackHandler could not be found.
    at org.apache.kafka.common.config.ConfigDef.parseType(ConfigDef.java:758)
    at org.apache.kafka.common.config.ConfigDef.parseValue(ConfigDef.java:504)
    at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:497)
    at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:113)
    at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:133)
    at org.apache.kafka.clients.consumer.ConsumerConfig.<init>(ConsumerConfig.java:630)
    at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:664)
    at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:645)
    at kafka.tools.ConsoleConsumer$.run(ConsoleConsumer.scala:65)
    at kafka.tools.ConsoleConsumer$.main(ConsoleConsumer.scala:52)
    at kafka.tools.ConsoleConsumer.main(ConsoleConsumer.scala)

I saw it was noted in the troubleshooting and it says to:

That means that this aws-msk-iam-auth library is not on the classpath of the Kafka client. Please add the aws-msk-iam-auth library to the classpath and try again.

I tried to do it with this command below: my command

env CLASSPATH=/Users/tzookb/kafka-connect/aws-msk-iam-auth-1.1.1-all.jar bash kafka-avro-console-consumer \
   --topic mytopic \
   --bootstrap-server "myserver.com:9098,myserver2.com:9098,myserver3.com:9098" \
   --from-beginning \
   --consumer.config /Users/tzookb/kafka-connect/service-desk-gamma-client.properties \
   --property schema.registry.url=http://localhost:8080/schemaRegistry \
   --property print.key=true \
   --property print.value=true \
   --value-deserializer io.confluent.kafka.serializers.KafkaAvroDeserializer \
   --key-deserializer org.apache.kafka.common.serialization.StringDeserializer

but I still get the same error. anything Im missing?

[Pablo-platypus]

Depending on how you are running your consumer, you need to add the aws-msk-iam-auth library to the classpath differently. Some options are:

(1) add it to the project dependencies using for example the maven repo in the ReadMe then build the project. (2) add it to the java env classpath like you are doing in your example. (3) add it to external jar files manifest files.

About (3), running an already existing jar file will probably override the env classpath with the internals META-INF/MANIFEST.mf file class-path, so you need to edit it to let it know where to find this external jar library.

From https://javarevisited.blogspot.com/2011/01/how-classpath-work-in-java.html#axzz7qlxl0T5c In this case, you can set your Java classpath in the META-INF/MANIFEST.MF file by using the Class-Path attribute. In short Class-path attribute in manifest file overrides classpath specified by -cp, -classpath or CLASSPATH environment variable.

How to edit an existing jar file's MANIFEST: https://stackoverflow.com/questions/22682809/how-can-i-change-manifest-file-in-java

1. Extract the manifest: jar xvf MyProject.jar META-INF/MANIFEST.MF

2. Edit the manifest

3. Reinsert the edited manifest: jar uvf MyProject.jar META-INF/MANIFEST.MF

For 3. I actually use (add the M option) jar uMvf MyProject.jar META-INF/MANIFEST.MF

[jbarrow6]

I was able to solve this issue by copying aws-msk-iam-auth-1.1.1-all.jar into kafka_2.12-2.8.1/libs/

[jyoti-c]

I was able to solve this issue by copying aws-msk-iam-auth-1.1.1-all.jar into kafka_2.12-2.8.1/libs/

thanks, worked for me too

[singhbaljit]

It doesn't seem to be working using kafka_2.13-2.8.2; works with kafka_2.12-2.8.2.

[itconsultor]

Thanks, it worked for me too

[Devarsh23]

We are facing the same issue when we are trying to use the IAM authentication from kafka connect. We are getting Invalid value software.amazon.msk.auth.iam.IAMClientCallbackHandler for configuration sasl.client.callback.handler.class: Class software.amazon.msk.auth.iam.IAMClientCallbackHandler could not be found.

We are using the confluentinc/cp-kafka-connect-base image with version 7.5.0. So, while creating our image, which will use the IAM auth, we are using the latest jar of version 2.0.3. In this image, we are copying this jar into the following path

1. /usr/share/java/
2. /etc-kafka-connect/jars
3. /usr/share/java/cp-base-new

We are running our kafka-connect in the distributed mode. We have also set the CLASSPATH in the docker file as an environment variable to this path /usr/share/java/. Also, we have set the plugin path for our kafka-connect to /usr/share/java,/usr/share/confluent-hub-components But after all this, we still get the same error mentioned above. Do you guys have any idea on how to resolve this issue?

[idanna]

@Devarsh23 Doing the following while building the connect image worked for me:

FROM confluentinc/cp-kafka-connect-base:latest
COPY aws-msk-iam-auth-2.0.3-all.jar /usr/share/java/kafka