aws / aws-msk-iam-auth

Enables developers to use AWS Identity and Access Management (IAM) to connect to their Amazon Managed Streaming for Apache Kafka (Amazon MSK) clusters.
Apache License 2.0
145 stars 67 forks source link

Kafka Connect Unable to connect to AWS MSK after advertised port aleration #79

Closed ma-kadri closed 11 months ago

ma-kadri commented 2 years ago

I have a AWS MSK running on 2.6.2 with IAM auth, thing works well, untill i tried implementing a gold man sachs architecture which i saw here.

I am doing the Single NLB setup, which involves altering the Advertised port. The command i used is as follows

./kafka-configs \
--bootstrap-server $B1:9098 \
--entity-type brokers \
--entity-name 1 \
--alter \
--command-config client_iam.properties \
--add-config advertised.listeners=[CLIENT_IAM://$B1:9001,REPLICATION://b-1-internal.$KF_DOMAIN:9093,REPLICATION_SECURE://b-1-internal.$KF_DOMAIN:9095]

After this, although i am am able to produce and consume data via a EC2 in the same subnet as MSK, The Snowflake Connector which was running prior to running above command, started failing. I am just lost what more needs to be done here to make the connector work again, any help is appreciated.

connector.class=com.snowflake.kafka.connector.SnowflakeSinkConnector
snowflake.topic2table.map=boosterflow:boosterflow
tasks.max=8
topics=boosterflow
snowflake.topic.sasl.client.callback.handler.class=software.amazon.msk.auth.iam.IAMClientCallbackHandler
buffer.flush.time=60
snowflake.url.name=XXXX.snowflakecomputing.com:443
snowflake.database.name=XXXXX
snowflake.schema.name=XXXXXX
buffer.count.records=10000
snowflake.user.name=XXXXX
snowflake.private.key="XXXXXXX"
snowflake.topic.sasl.jaas.config=software.amazon.msk.auth.iam.IAMLoginModule required;
snowflake.topic.security.protocol=SASL_SSL
value.converter=com.snowflake.kafka.connector.records.SnowflakeJsonConverter
snowflake.topic.sasl.mechanism=AWS_MSK_IAM
buffer.size.bytes=5000000

I am seeing errors like these in connector

Worker-03e3eee9d7f02cfe6] [2022-09-28 16:52:43,013] WARN [AdminClient clientId=adminclient-8] Connection to node 2 (b-2.XXX.XXX.c16.kafka.us-east-1.amazonaws.com/INTERNAL_IP) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient:782)
[Worker-03e3eee9d7f02cfe6] [2022-09-28 16:52:44,620] INFO [Worker clientId=connect-1, groupId=__amazon_msk_connect_cluster_XXX-json_631e64b8-7451-4c20-aabf-6a5fb80081a6-4] Reading to end of config log; current config state offset: 12 (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1198)
[Worker-03e3eee9d7f02cfe6] [2022-09-28 16:52:47,620] WARN [Worker clientId=connect-1, groupId=__amazon_msk_connect_cluster_XXX-json_631e64b8-7451-4c20-aabf-6a5fb80081a6-4] Didn't reach end of config log quickly enough (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1209)
[Worker-03e3eee9d7f02cfe6] java.util.concurrent.TimeoutException: Timed out waiting for future
github-actions[bot] commented 1 year ago

We have noticed this issue has not received attention in 1 year. We will close this issue for now. If you think this is in error, please feel free to comment and reopen the issue.