Exception while evaluating challenge [Caused by javax.security.auth.callback.UnsupportedCallbackException: Unrecognized SASL ClientCallback]

[miltonhit]

Hello,

I'm trying to make a producer in java with this code:

Properties props = new Properties(); props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG,KAFKA_BOOTSTRAP); props.put(ProducerConfig.CLIENT_ID_CONFIG, "KafkaExampleProducer"); props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, LongSerializer.class.getName()); props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName()); props.put("security.protocol", "SASL_SSL"); props.put("sasl.mechanism", "AWS_MSK_IAM"); props.put("sasl.jaas.config", "software.amazon.msk.auth.iam.IAMLoginModule required;"); props.put("sasl.client.callback.handler.class", "software.amazon.msk.auth.iam.IAMClientCallbackHandler"); return new KafkaProducer<>(props);

And I'm getting this error:

Caused by: javax.security.sasl.SaslException: Exception while evaluating challenge [Caused by javax.security.auth.callback.UnsupportedCallbackException: Unrecognized SASL ClientCallback] at software.amazon.msk.auth.iam.internals.IAMSaslClient.evaluateChallenge(IAMSaslClient.java:113) at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator$2.run(SaslClientAuthenticator.java:278) at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator$2.run(SaslClientAuthenticator.java:276) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.createSaslToken(SaslClientAuthenticator.java:276)

I already imported the libs into the gradle:

software.amazon.msk:aws-msk-iam-auth:1.0.0

And my kafka-cluster was created using the IAM auth configuration.... I am running this code snippet on an ec2 that has as a default role the necessary policies to access the cluster.

I'm not able to find anything on the internet talking about this error. It seems to me something about configuration and not connectivity.

Can you help me? :(

[sayantacC]

This is puzzling and likely indeed configuration related.

It looks like the IAMClientCallbackHandler set by sasl.client.callback.handler.class is not being invoked and instead the default SaslClientCallbackHandler is being invoked at

https://github.com/apache/kafka/blob/13ffebe2f1012da473189c4fe86fd14b83809962/clients/src/main/java/org/apache/kafka/common/security/authenticator/SaslClientCallbackHandler.java#L98

This likely means that the property for sasl.client.callback.handler.class is either not being properly set or being overridden.

1. Is there a possibility that is happening ?
2. Would it be possible for you to dump out the configurations that the producer is actually getting or get it from a debugger ? For example, even a stray space in front of the key could lead to such a misconfiguration.
3. Also, which version of the kafka clients library are you using ?

[grsubramanian]

@miltonhit were you able to find a solution to this problem?

[grsubramanian]

@miltonhit we have seen instances of similar issues happening when deploying Kafka apps via systems such as Flink that shade and relocate classes. Can you describe how you are building and deploying your application?

[sanjoymukherjee2019]

I faced a similar situation when I tried to use spring boot with IAM auth for msk. I managed to make it work with the following settings in application.yaml. The EC2 instance had appropriate roles to create topic etc and the security group allowed access to zk and kafka. Hope this helps application.yaml.zip

[siagroove]

We are facing a similar issue when we run Debezium MYSQL which is a Kafka Connector.

[2021-08-19 16:33:00,272] INFO [Producer clientId=connector-producer-test-cdc-iam-msk-0] Failed authentication with BROKER (An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: Exception while evaluating challenge [Caused by javax.security.auth.callback.UnsupportedCallbackException: Unsupported callback type:software.amazon.msk.auth.iam.internals.AWSCredentialsCallback]) occurred when evaluating SASL token received from the Kafka Broker. Kafka Client will go to AUTHENTICATION_FAILED state.) (org.apache.kafka.common.network.Selector:616)
[2021-08-19 16:33:00,273] ERROR [Producer clientId=connector-producer-test-cdc-iam-msk-0] Connection to node -2 (BROKER) failed authentication due to: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: Exception while evaluating challenge [Caused by javax.security.auth.callback.UnsupportedCallbackException: Unsupported callback type:software.amazon.msk.auth.iam.internals.AWSCredentialsCallback]) occurred when evaluating SASL token received from the Kafka Broker. Kafka Client will go to AUTHENTICATION_FAILED state. (org.apache.kafka.clients.NetworkClient:765)

Has anyone found a solution?

[sayantacC]

@siagroove Is the aws-msk-iam-auth library being shaded or relocated in your use case ? Based on the error message, that is the most likely cause that the AWSCredentialsCallback is being detected as unsupported. Would it be possible for you to turn on debug logging for aws-msk-iam-auth ?

[siagroove]

@sayantacC I am using the JAR not the pom dependency. We're adding all the debezium connector jars plus the iam auth jar to our docker container and then running the connect task. I believe that it detects the jar because when I put it in the wrong path it complains. We also add the jar path to our plugin.path in connect properties. I enabled DEBUG logs and attached them here. Let me know if you need more logs. logs.log

[siagroove]

@sayantacC any thoughts on the above?

[sayantacC]

@siagroove Thanks for the logs and I apologize for the delay. From the logs you provided, my suspicion is that this might be a problem with class loaders. I have added more detailed logging and error message in aws-msk-iam-auth. It should tell us if IAMClientCallbackHandler and AWSCredentialsCallback are being loaded by different classloaders or being relocated. Would it be possible for you to run with the latest version of the library with debug log and provide the logs ? I will try to respond faster this time.

[siagroove]

@sayantacC thanks for your reply. How can I run the latest library? we use the JAR's, have you released a JAR for this change?

[sayantacC]

I have checked the code in. Could you build the uber jar using the instructions at https://github.com/aws/aws-msk-iam-auth#building-from-source ? This will build a jar containing all the dependencies of the library.

[siagroove]

@sayantacC here are the new logs. Let me know if you need anything else. logs.log

[sayantacC]

@siagroove I am really sorry but these logs do not seem to have been generated from the latest version of the library. Could you please verify that you pulled and built the latest version from github ? The logs should contain one or more lines similar to:

class: software.amazon.msk.auth.iam.internals.AWSCredentialsCallback classloader:

[siagroove]

@sayantacC I'm sorry, I attached the wrong log file. Here is the correct one. latest logs.log

[sayantacC]

@siagroove The problem, as suspected, is that the AWSCredentialsCallback is being loaded by a different class loader than the IAMClientCallbackHandler. From the logs:

Unsupported callback type: class: software.amazon.msk.auth.iam.internals.AWSCredentialsCallback classloader: jdk.internal.loader.ClassLoaders$AppClassLoader@75b84c92 from class: software.amazon.msk.auth.iam.IAMClientCallbackHandler classloader: PluginClassLoader{pluginLocation=file:/usr/share/connect/debezium/

The AWSCredentialsCallback is being loaded by the jdk.internal.loader.ClassLoaders$AppClassLoader whereas IAMClientCallbackHandler is being loaded by the PluginClassLoader that ships with KafkaConnect. I think this might be happening because the aws-msk-iam-auth library has been placed in the plugin path.

Since the aws-msk-iam-authlibrary is actually used by the Kafka producer and consumer clients and not the Kafka Connect plugin itself, it should be placed outside the plugin path. Could you move the aws-msk-iam-auth library away from the plugin path and into some other location that is still on the classpath ?

[siagroove]

@sayantacC you were correct. That fixed our issue! thank you so much.

[dwang2]

@sayantacC I've been encountering similar exceptions even though the class loader appears correct,

"[2021-09-02 03:38:20,288] DEBUG Type information for callback: class: software.amazon.msk.auth.iam.internals.AWSCredentialsCallback classloader: jdk.internal.loader.ClassLoaders$AppClassLoader@9e89d68 from class: software.amazon.msk.auth.iam.IAMClientCallbackHandler classloader: jdk.internal.loader.ClassLoaders$AppClassLoader@9e89d68 (software.amazon.msk.auth.iam.IAMClientCallbackHandler)"

2021-09-02 03:38:31,085] ERROR [Producer clientId=connector-producer-clientmgt-source-clnt-0] Connection to node -2 (b-1.kafka-dev-1.fk54o5.c17.kafka.us-east-1.amazonaws.com/10.10.128.20:9098) failed authentication due to: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: Exception while evaluating challenge [Caused by javax.security.auth.callback.UnsupportedCallbackException: Unrecognized SASL ClientCallback]) occurred when evaluating SASL token received from the Kafka Broker. Kafka Client will go to AUTHENTICATION_FAILED state. (org.apache.kafka.clients.NetworkClient)

kafka-connect.log

[sayantacC]

@dwang2 From the log file, it seems that some components do succeed in connecting using IAM:

[2021-09-02 03:38:24,696] DEBUG State COMPLETE at end of evaluating challenge (software.amazon.msk.auth.iam.internals.IAMSaslClient)

As far as I can tell, these are Producers with clientIds of the form : [Producer clientId=producer-2].

The ones that fail seem to be from another component with different clientIds: [Producer clientId=connector-producer-clientmgt-source-clnt-0] or [Producer clientId=connector-producer-clientmgt-source-tnnt-0].

From Unrecognized SASL ClientCallback it seems that these producer clients have not been configured for IAM. Is it possible that these producers are receiving different configurations from the earlier ones ?

It seems likely as I see some configuration with sasl.client.callback.handler.class set to null instead of IAMClientCallbackHandler:

[2021-09-02 03:38:30,526] INFO ProducerConfig values: 
    acks = -1
    batch.size = 16384
    bootstrap.servers = [SASL_SSL://b-3.kafka-dev-1.fk54o5.c17.kafka.us-east-1.amazonaws.com:9098, SASL_SSL://b-1.kafka-dev-1.fk54o5.c17.kafka.us-east-1.amazonaws.com:9098, SASL_SSL://b-2.kafka-dev-1.fk54o5.c17.kafka.us-east-1.amazonaws.com:9098]
    buffer.memory = 33554432
    client.dns.lookup = use_all_dns_ips
    client.id = connector-producer-clientmgt-source-tnnt-0
    compression.type = none
    connections.max.idle.ms = 540000
    delivery.timeout.ms = 2147483647
    enable.idempotence = false
    interceptor.classes = [io.confluent.monitoring.clients.interceptor.MonitoringProducerInterceptor]
    internal.auto.downgrade.txn.commit = false
    key.serializer = class org.apache.kafka.common.serialization.ByteArraySerializer
    linger.ms = 0
    max.block.ms = 9223372036854775807
    max.in.flight.requests.per.connection = 1
    max.request.size = 1048576
    metadata.max.age.ms = 300000
    metadata.max.idle.ms = 300000
    metric.reporters = []
    metrics.num.samples = 2
    metrics.recording.level = INFO
    metrics.sample.window.ms = 30000
    partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner
    receive.buffer.bytes = 32768
    reconnect.backoff.max.ms = 1000
    reconnect.backoff.ms = 50
    request.timeout.ms = 2147483647
    retries = 2147483647
    retry.backoff.ms = 100
    sasl.client.callback.handler.class = null
    sasl.jaas.config = [hidden]
    sasl.kerberos.kinit.cmd = /usr/bin/kinit
    sasl.kerberos.min.time.before.relogin = 60000
    sasl.kerberos.service.name = null
    sasl.kerberos.ticket.renew.jitter = 0.05
    sasl.kerberos.ticket.renew.window.factor = 0.8
    sasl.login.callback.handler.class = null
    sasl.login.class = null
    sasl.login.refresh.buffer.seconds = 300
    sasl.login.refresh.min.period.seconds = 60
    sasl.login.refresh.window.factor = 0.8
    sasl.login.refresh.window.jitter = 0.05
    sasl.mechanism = AWS_MSK_IAM
    security.protocol = SASL_SSL
    security.providers = null
    send.buffer.bytes = 131072
    ssl.cipher.suites = null
    ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
    ssl.endpoint.identification.algorithm = https
    ssl.engine.factory.class = null
    ssl.key.password = null
    ssl.keymanager.algorithm = SunX509
    ssl.keystore.location = null
    ssl.keystore.password = null
    ssl.keystore.type = JKS
    ssl.protocol = TLSv1.3
    ssl.provider = null
    ssl.secure.random.implementation = null
    ssl.trustmanager.algorithm = PKIX
    ssl.truststore.location = null
    ssl.truststore.password = null
    ssl.truststore.type = JKS
    transaction.timeout.ms = 60000
    transactional.id = null
    value.serializer = class org.apache.kafka.common.serialization.ByteArraySerializer
 (org.apache.kafka.clients.producer.ProducerConfig)

[dwang2]

@sayantacC You're right, the confluent product -"kafka-connect" referred above requires seperate kafka configurations for producer clients and consumer clients. I ended up adding the below configurations to the application.

consumer.sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required; consumer.sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler producer.sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required; producer.sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler

That fixed the issue. Thank you very much!

[sayantacC]

Please see Troubleshooting section in README for steps to check if different class loaders are being used for different classes in aws-msk-iam-auth.

Original question has not been active. Subsequent questions have been answered. Closing issue.

[bipul-git]

Hi @sayantacC I'm facing the same issue while starting kafka consumer Below is my configuration

ssl.truststore.location=/tmp/kafka.client.truststore.jks
security.protocol=SASL_SSL
sasl.mechanism=AWS_MSK_IAM
sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler
bootstrap.servers=SERVER_LIST

But the classloaders for AWSCredentialsCallback and IAMClientCallbackHandler are both same i.e. ParallelWebappClassLoader

org.apache.kafka.common.errors.SaslAuthenticationException: An error: (java.security.PrivilegedActionException: javax.security.sasl.SaslException: Exception while evaluating challenge [Caused by javax.security.auth.callback.UnsupportedCallbackException: Unsupported callback type: class: software.amazon.msk.auth.iam.internals.AWSCredentialsCallback classloader: ParallelWebappClassLoader
  context: Unknown
  delegate: false
----------> Parent Classloader:
java.net.URLClassLoader@27d6c5e0
 from class: software.amazon.msk.auth.iam.IAMClientCallbackHandler classloader: ParallelWebappClassLoader
  context: efm
  delegate: false
----------> Parent Classloader:
java.net.URLClassLoader@27d6c5e0