search

aws / aws-msk-iam-auth

Enables developers to use AWS Identity and Access Management (IAM) to connect to their Amazon Managed Streaming for Apache Kafka (Amazon MSK) clusters.
Apache License 2.0
137 stars 65 forks source link

Databricks - StreamingQueryException #115

Open nph4rd opened 1 year ago

nph4rd commented 1 year ago

Hi - I'm trying to read from Kafka from Databricks, as outlined here.

At my organisation we can only connect via IAM, so I added the following options:

  .option("kafka.security.protocol", "SASL_SSL")
  .option("kafka.sasl.mechanism", "AWS_MSK_IAM")
  .option("kafka.sasl.jaas.config", "software.amazon.msk.auth.iam.IAMLoginModule required;")
  .option("kafka.sasl.client.callback.handler.class", "software.amazon.msk.auth.iam.IAMClientCallbackHandler")

and uploaded the JAR file for the 1.1.6 version of aws-msk-iam-auth to Databricks.

This resulted in the following error message:

StreamingQueryException: Query[...] terminated with exception: org/apache/kafka/common/security/auth/AuthenticateCallbackHandler

I found a potential patch here.

However, it would be far from ideal to work with a patched forked of this library. Is there anything I can do on my side to make the connection work?