search

aws / aws-msk-iam-auth

Enables developers to use AWS Identity and Access Management (IAM) to connect to their Amazon Managed Streaming for Apache Kafka (Amazon MSK) clusters.
Apache License 2.0
145 stars 67 forks source link

IAMClientCallbackHandler could not be found #96

Open tzookb opened 1 year ago

tzookb commented 1 year ago

keep getting this error:

[2023-01-06 14:13:05,494] ERROR Unknown error when running consumer:  (kafka.tools.ConsoleConsumer$:43)
org.apache.kafka.common.config.ConfigException: Invalid value software.amazon.msk.auth.iam.IAMClientCallbackHandler for configuration sasl.client.callback.handler.class: Class software.amazon.msk.auth.iam.IAMClientCallbackHandler could not be found.
    at org.apache.kafka.common.config.ConfigDef.parseType(ConfigDef.java:758)
    at org.apache.kafka.common.config.ConfigDef.parseValue(ConfigDef.java:504)
    at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:497)
    at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:113)
    at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:133)
    at org.apache.kafka.clients.consumer.ConsumerConfig.<init>(ConsumerConfig.java:630)
    at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:664)
    at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:645)
    at kafka.tools.ConsoleConsumer$.run(ConsoleConsumer.scala:65)
    at kafka.tools.ConsoleConsumer$.main(ConsoleConsumer.scala:52)
    at kafka.tools.ConsoleConsumer.main(ConsoleConsumer.scala)

I saw it was noted in the troubleshooting and it says to:

That means that this aws-msk-iam-auth library is not on the classpath of the Kafka client. Please add the aws-msk-iam-auth library to the classpath and try again.

I tried to do it with this command below: my command

env CLASSPATH=/Users/tzookb/kafka-connect/aws-msk-iam-auth-1.1.1-all.jar bash kafka-avro-console-consumer \
   --topic mytopic \
   --bootstrap-server "myserver.com:9098,myserver2.com:9098,myserver3.com:9098" \
   --from-beginning \
   --consumer.config /Users/tzookb/kafka-connect/service-desk-gamma-client.properties \
   --property schema.registry.url=http://localhost:8080/schemaRegistry \
   --property print.key=true \
   --property print.value=true \
   --value-deserializer io.confluent.kafka.serializers.KafkaAvroDeserializer \
   --key-deserializer org.apache.kafka.common.serialization.StringDeserializer

but I still get the same error. anything Im missing?

Pablo-platypus commented 1 year ago

Depending on how you are running your consumer, you need to add the aws-msk-iam-auth library to the classpath differently. Some options are:

(1) add it to the project dependencies using for example the maven repo in the ReadMe then build the project. (2) add it to the java env classpath like you are doing in your example. (3) add it to external jar files manifest files.

About (3), running an already existing jar file will probably override the env classpath with the internals META-INF/MANIFEST.mf file class-path, so you need to edit it to let it know where to find this external jar library.

From https://javarevisited.blogspot.com/2011/01/how-classpath-work-in-java.html#axzz7qlxl0T5c In this case, you can set your Java classpath in the META-INF/MANIFEST.MF file by using the Class-Path attribute. In short Class-path attribute in manifest file overrides classpath specified by -cp, -classpath or CLASSPATH environment variable.

How to edit an existing jar file's MANIFEST: https://stackoverflow.com/questions/22682809/how-can-i-change-manifest-file-in-java

  1. Extract the manifest: jar xvf MyProject.jar META-INF/MANIFEST.MF

  2. Edit the manifest

  3. Reinsert the edited manifest: jar uvf MyProject.jar META-INF/MANIFEST.MF

For 3. I actually use (add the M option) jar uMvf MyProject.jar META-INF/MANIFEST.MF

jbarrow6 commented 1 year ago

I was able to solve this issue by copying aws-msk-iam-auth-1.1.1-all.jar into kafka_2.12-2.8.1/libs/

jyoti-c commented 1 year ago

I was able to solve this issue by copying aws-msk-iam-auth-1.1.1-all.jar into kafka_2.12-2.8.1/libs/

thanks, worked for me too

singhbaljit commented 1 year ago

It doesn't seem to be working using kafka_2.13-2.8.2; works with kafka_2.12-2.8.2.

itconsultor commented 1 year ago

Thanks, it worked for me too

Devarsh23 commented 9 months ago

We are facing the same issue when we are trying to use the IAM authentication from kafka connect. We are getting Invalid value software.amazon.msk.auth.iam.IAMClientCallbackHandler for configuration sasl.client.callback.handler.class: Class software.amazon.msk.auth.iam.IAMClientCallbackHandler could not be found.

We are using the confluentinc/cp-kafka-connect-base image with version 7.5.0. So, while creating our image, which will use the IAM auth, we are using the latest jar of version 2.0.3. In this image, we are copying this jar into the following path

  1. /usr/share/java/
  2. /etc-kafka-connect/jars
  3. /usr/share/java/cp-base-new

We are running our kafka-connect in the distributed mode. We have also set the CLASSPATH in the docker file as an environment variable to this path /usr/share/java/. Also, we have set the plugin path for our kafka-connect to /usr/share/java,/usr/share/confluent-hub-components But after all this, we still get the same error mentioned above. Do you guys have any idea on how to resolve this issue?

idanna commented 8 months ago

@Devarsh23 Doing the following while building the connect image worked for me: 

FROM confluentinc/cp-kafka-connect-base:latest
COPY aws-msk-iam-auth-2.0.3-all.jar /usr/share/java/kafka
sabper commented 3 months ago

we are using the confluentinc/cp-kafka-connect-base image with version 7.7.0.

following work for me:

DGondhiya commented 3 months ago

we have tried this path (/usr/share/java/cp-base-new/) in distroless and scratch image but getting the same error. Any lead on classpath, where can we put this explicitly?

yanivkrol commented 3 months ago

Hi, I'm trying to use kafka cli with AWS_MSK_IAM mechanism and I get the same error.

I put the jar aws-msk-iam-auth-2.2.0-all.jar in /opt/homebrew/Cellar/kafka/3.7.1/libexec/libsz.

Any ideas?

command: kafka-topics --bootstrap-server server --describe --topic topic --command-config client.properties

Also doesn't work If install kafka directly :

curl https://archive.apache.org/dist/kafka/2.8.1/kafka_2.12-2.8.1.tgz -o kafka_2.12-2.8.1.tgz

curl https://github.com/aws/aws-msk-iam-auth/releases/download/v1.1.1/aws-msk-iam-auth-1.1.1-all.jar -o aws-msk-iam-auth-1.1.1-all.jar

cp aws-msk-iam-auth-1.1.1-all.jar kafka_2.12-2.8.1/libs/

kafka_2.12-2.8.1/bin/kafka-topics.sh --list

Exception in thread "main" org.apache.kafka.common.config.ConfigException: Invalid value
software.amazon.msk.auth.iam.IAMClientCallbackHandler for configuration sasl.client.callback.handler.class: 
Class software.amazon.msk.auth.iam.IAMClientCallbackHandler could not be found.
kvanbrabant commented 3 months ago

We faced the same error when converting our kafka-connect cluster from TLS authentication to AWS IAM authentication. In the end, we got it working by copying the jar file in these locations:

AFAIK, the first location is where by default the classpath gets loaded for creating the admin client, serving the connector list, status, tasks, etc. through the REST endpoint. This admin client starts by reading the deployed connectors from a kafka topic and needs read access to the topic storing these connectors, so it fails when it can't authenticate. The second location is probably added to the classpath of each connector plugin. This is needed when a sink connector needs to read from a topic or a source connector needs to write to it. Also, sink connectors need to be allowed to use the group called "connector-group" (by default). I can't find any documentation on confluent that acknowledges this, so if anyone knows where it is, please post the link.

thranasingha commented 2 months ago

We faced the same issue when we were trying to create kafka consumer from spring boot application with plain java. We needed it to be a async task. So we used a following code snippet.

CompletableFuture.runAsync( () -> { ... });

With this asynchronus task, it will try to instantiate software.amazon.msk.auth.iam.IAMLoginModule with the class loader. Class loader which will be used for CompletableFuture.runAsync() is ForkJoinPool which is created when JVM is initialized. So it will not able to access the spring class loader which leads to have class not found exception.

Solution we found is to provide a separate executor pool which suggested in https://stackoverflow.com/questions/49113207/completablefuture-forkjoinpool-set-class-loader answer

ExecutorService pool = Executors.newFixedThreadPool(10); final CompletableFuture<String> future = CompletableFuture.supplyAsync(() -> { /* ... */ }, pool);

This executor pool solved my class loader issue.