MSK AWS connect

[clazalde]

Im trying to set the connector in AWS connect with a MSK serverless cluster in AWS but I keep getting the error:

[Worker-016e5f49a1715bc84] [2023-05-10 17:40:33,607] WARN [iceberg-sink-connector\|task-0] Unable to retrieve the requested metadata. (software.amazon.awssdk.regions.internal.util.EC2MetadataUtils:398) -- [Worker-016e5f49a1715bc84] [2023-05-10 17:40:33,608] ERROR [iceberg-sink-connector\|task-0] WorkerSinkTask{id=iceberg-sink-connector-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask:191) [Worker-016e5f49a1715bc84] software.amazon.awssdk.core.exception.SdkClientException: Unable to load region from any of the providers in the chain software.amazon.awssdk.regions.providers.DefaultAwsRegionProviderChain@c649ef2: [software.amazon.awssdk.regions.providers.SystemSettingsRegionProvider@52410e10: Unable to load region from system settings. Region must be specified either via environment variable (AWS_REGION) or system property (aws.region)., software.amazon.awssdk.regions.providers.AwsProfileRegionProvider@2d28feba: No region provided in profile: default, software.amazon.awssdk.regions.providers.InstanceProfileRegionProvider@1c19d17a: Unable to retrieve region information from EC2 Metadata service. Please make sure the application is running on EC2.]

Any suggestion on how to declare region in the connector?

[clazalde]

this is my config by the way:

"connector.class"= "com.getindata.kafka.connect.iceberg.sink.IcebergSink"
"tasks.max"= "1"
"topics"= "sqa-sdiv-data-warehouse.public.test_2,sqa-sdiv-data-warehouse.public.test_table"

"upsert"= true
"upsert.keep-deletes"= true

"table.auto-create"= true
"table.write-format"= "parquet"
"table.namespace"= "sdiv"
"table.prefix"= "cdc_"

"iceberg.catalog-impl"= "org.apache.iceberg.aws.glue.GlueCatalog"
"iceberg.fs.defaultFS"= "s3a://sqa-subdiv-kafka-iceberg-poc/sdiv-iceberg"
"iceberg.warehouse"= "s3a://sqa-subdiv-kafka-iceberg-poc/sdiv-iceberg"
"iceberg.fs.s3a.endpoint.region" = "us-west-2"
"iceberg.fs.s3a.endpoint" = "s3.us-west-2.amazonaws.com"
"iceberg.com.amazonaws.services.s3.enableV4"= true
"iceberg.com.amazonaws.services.s3a.enableV4"= true
"iceberg.fs.s3a.aws.credentials.provider"= "com.amazonaws.auth.DefaultAWSCredentialsProviderChain"
"iceberg.fs.s3a.path.style.access"= true
"iceberg.fs.s3a.impl"= "org.apache.hadoop.fs.s3a.S3AFileSystem"

[glongrais]

I am having the exact same issue. Unable to specify the region to the worker in any way

[rj-u-developer]

Facing the same issue.

[clazalde]

Solved by modifying the connector and compiling it again, tomorrow will share the modified code so you can do the same

[rj-u-developer]

Ok, it will be very helpful. waiting for updated connector.

[rj-u-developer]

@clazalde , I would appreciate if you can share the updated connector. Thanks in advance.

[clazalde]

In the public void start(Map<String, String> properties) { Add the following: System.setProperty("aws.region", "us-west-2"); Replace it with your region

[rj-u-developer]

@clazalde , I have set the aws.region. Now I am getting below error:-

Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted. Error: Unable to execute HTTP request: connect timed out (org.apache.kafka.connect.runtime.WorkerSinkTask:612)

[sagarm-traveloka]

I am also facing the same issue while trying to use this connector in AWS MSK cluster. Any input would be greatly appreciated. 🙏 software.amazon.awssdk.core.exception.SdkClientException: Unable to load region from any of the providers in the chain