Closed NewtonXu closed 1 year ago
@NewtonXu Can you avoid adding aws jars. hudi-aws bundle contains the required aws jars also. These should be enough ideally - org.apache.hudi:hudi-spark3.3-bundle_2.12:0.13.1 org.apache.hudi:hudi-aws:0.13.1
@NewtonXu Were you able to get it working?
Yes, this worked for me. Thanks!
Describe the problem you faced
Trying to enable DynamoDB concurrency controls but unable to instantiate due to builder error.
To Reproduce
Steps to reproduce the behavior:
Executing the Spark job with these packages:
Enabling concurrency control with these settings
This job is created on AWS EMR Serverless
Expected behavior
DynamoDB lock is created
Environment Description
Hudi version : 13.1
Spark version : Spark version 3.3.1-amzn-0
Hive version :
Hadoop version :
Storage (HDFS/S3/GCS..) : S3
Running on Docker? (yes/no) : Yes (emr serverless)
Additional context
I'm running this job on Amazon EMR serverless, but I've decided to use the open-source Hudi bundle instead. Previously when using the AWS bundle, it could not find DynamoDBBasedLockProvider.
Stacktrace
Add the stacktrace of the error.
I am wondering if there is some incompatibility with the SDK's I've chosen? org.apache.hudi:hudi-spark3.3-bundle_2.12:0.13.1 com.amazonaws:dynamodb-lock-client:1.2.0 com.amazonaws:aws-java-sdk-dynamodb:1.12.490 com.amazonaws:aws-java-sdk-core:1.12.490 org.apache.hudi:hudi-aws:0.13.1