search

apache / hudi

Upserts, Deletes And Incremental Processing on Big Data.
https://hudi.apache.org/
Apache License 2.0
5.48k stars 2.43k forks source link

[SUPPORT] Hudi Sink Connector shows broker disconnected #9070

Open ennox108 opened 1 year ago

ennox108 commented 1 year ago

I am trying to add the Hudi Sink Connector to AWS MSK using the below config

connector.class=org.apache.hudi.connect.HoodieSinkConnector s3.region=us-east-1 hoodie.base.path=s3://data_bucket hoodie.table.type=MERGE_ON_READ flush.size=5 tasks.max=4 topics=** hoodie.schemaprovider.class=org.apache.hudi.schema.SchemaRegistryProvider bootstrap.servers=**** hoodie.datasource.write.partitionpath.field=** value.converter.region=us-east-1 hoodie.kafka.commit.interval.secs=60 hoodie.table.name=hudi-test hoodie.datasource.write.recordkey.field=*** value.converter.schemas.enable=false value.converter=org.apache.kafka.connect.json.JsonConverter key.converter=org.apache.kafka.connect.storage.StringConverter

The connector status shows running but theres is no data that getting written to s3. When i check the logs it shows the below error

Bootstrap broker **** (id: -2 rack: null) disconnected

I have verified if the brokers are in a healthy state. I have also tested other sink connectors using similar config and those connectors are correctly producing data to s3.

ad1happy2go commented 1 year ago

@ennox108 What hudi version you are using? Can you give complete error stack trace what you are getting in Kafka connect logs?

ennox108 commented 1 year ago

@ad1happy2go I am using hudi 0.11

I see same error as below for all the 3 brokers image

ad1happy2go commented 1 year ago

@ennox108 The configs looks okay but not sure if we have ever tested Kafka connect with AWS MSK. Can you check if the steps you followed works for you on vanilla Kafka? Also, Did you tried hudi version 0.13.1 or 0.12.3 ?

We need to reproduce and test it with AWS MSK.

ennox108 commented 1 year ago

@ad1happy2go tried using 0.13.1 as well as 0.12.3. Still facing the same issue.

ennox108 commented 1 year ago

@ad1happy2go the connector was able to create a folder. However, its still not writing any data to the folder. I am seeing the below error message image

ennox108 commented 1 year ago

@ad1happy2go any update on this?

parepallykiran commented 1 year ago

I'm also running into similar issue trying to launch Hudi Kafka Connect to consume from MSK cluster with TLS enabled brokers. Is there any work around to get around this issue?

prabodh1194 commented 6 months ago

i am facing same issue. i have searched around everywhere. what am I missing?

soumilshah1995 commented 6 months ago

why not use deltastreamer instead !

prabodh1194 commented 6 months ago

yeah. i just wanted to check out kafka-connect. got massively stuck on this issue :( . anyways, i think prefixing the props with consumer.override works well.

prabodh1194 commented 6 months ago

but still facing a bunch of issues in the java classpath.