Enables developers to use AWS Identity and Access Management (IAM) to connect to their Amazon Managed Streaming for Apache Kafka (Amazon MSK) clusters.
Apache License 2.0
145
stars
67
forks
source link
Kafka Connect Unable to connect to AWS MSK after advertised port aleration #79
After this, although i am am able to produce and consume data via a EC2 in the same subnet as MSK,
The Snowflake Connector which was running prior to running above command, started failing.
I am just lost what more needs to be done here to make the connector work again, any help is appreciated.
Worker-03e3eee9d7f02cfe6] [2022-09-28 16:52:43,013] WARN [AdminClient clientId=adminclient-8] Connection to node 2 (b-2.XXX.XXX.c16.kafka.us-east-1.amazonaws.com/INTERNAL_IP) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient:782)
[Worker-03e3eee9d7f02cfe6] [2022-09-28 16:52:44,620] INFO [Worker clientId=connect-1, groupId=__amazon_msk_connect_cluster_XXX-json_631e64b8-7451-4c20-aabf-6a5fb80081a6-4] Reading to end of config log; current config state offset: 12 (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1198)
[Worker-03e3eee9d7f02cfe6] [2022-09-28 16:52:47,620] WARN [Worker clientId=connect-1, groupId=__amazon_msk_connect_cluster_XXX-json_631e64b8-7451-4c20-aabf-6a5fb80081a6-4] Didn't reach end of config log quickly enough (org.apache.kafka.connect.runtime.distributed.DistributedHerder:1209)
[Worker-03e3eee9d7f02cfe6] java.util.concurrent.TimeoutException: Timed out waiting for future
We have noticed this issue has not received attention in 1 year. We will close this issue for now. If you think this is in error, please feel free to comment and reopen the issue.
I have a AWS MSK running on 2.6.2 with IAM auth, thing works well, untill i tried implementing a gold man sachs architecture which i saw here.
I am doing the Single NLB setup, which involves altering the Advertised port. The command i used is as follows
After this, although i am am able to produce and consume data via a EC2 in the same subnet as MSK, The Snowflake Connector which was running prior to running above command, started failing. I am just lost what more needs to be done here to make the connector work again, any help is appreciated.
I am seeing errors like these in connector