awslabs / emr-dynamodb-connector

Implementations of open source Apache Hadoop/Hive interfaces which allow for ingesting data from Amazon DynamoDB
Apache License 2.0
217 stars 135 forks source link

Upgrade AWS SDK Java from 1.x to 2.x #172

Closed kevnzhao closed 1 year ago

kevnzhao commented 1 year ago

Issue #, if available:

Description of changes:

NOTE This PR should be merged after #171 .

AWS SDK Java 1.x will enter maintenance mode soon. Upgrade to AWS SDK Java 2.x as recommended by AWS SDK team.

All unit test passed after this change. Test results are pasted as below.

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running org.apache.hadoop.dynamodb.write.WriteIopsCalculatorTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.586 sec
Running org.apache.hadoop.dynamodb.split.DynamoDBSplitGeneratorTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.115 sec
Running org.apache.hadoop.dynamodb.DynamoDBUtilTest
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.69 sec
Running org.apache.hadoop.dynamodb.test.GsonTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.95 sec
Running org.apache.hadoop.dynamodb.DynamoDBFibonacciRetryerTest
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 40.429 sec
Running org.apache.hadoop.dynamodb.util.ClusterTopologyNodeCapacityProviderTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.169 sec
Running org.apache.hadoop.dynamodb.util.RoundRobinYarnContainerAllocatorTest
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 sec
Running org.apache.hadoop.dynamodb.util.TaskCalculatorTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.045 sec
Running org.apache.hadoop.dynamodb.DynamoDBClientTest
Tests run: 18, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 124.523 sec
Running org.apache.hadoop.dynamodb.read.DynamoDBRecordReaderTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.123 sec
Running org.apache.hadoop.dynamodb.read.ReadIopsCalculatorTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.024 sec
Running org.apache.hadoop.dynamodb.read.PageResultMultiplexerTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.742 sec
Running org.apache.hadoop.dynamodb.preader.RateControllerTest
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 19.262 sec
Running org.apache.hadoop.dynamodb.preader.TokenBucketTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.009 sec
Running org.apache.hadoop.dynamodb.preader.ReadManagerTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.625 sec
Running org.apache.hadoop.dynamodb.preader.ScanRecordReadRequestTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.058 sec
Running org.apache.hadoop.dynamodb.DynamoDBItemWritableTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.177 sec

Results :

Tests run: 81, Failures: 0, Errors: 0, Skipped: 0

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running org.apache.hadoop.hive.dynamodb.util.HiveDynamoDBUtilTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.016 sec
Running org.apache.hadoop.hive.dynamodb.type.HiveDynamoDBTypeTest
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.214 sec
Running org.apache.hadoop.hive.dynamodb.DynamoDBObjectInspectorTest
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.005 sec
Running org.apache.hadoop.hive.dynamodb.DynamoDBStorageHandlerTest
Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.103 sec
Running org.apache.hadoop.hive.dynamodb.filter.DynamoDBFilterPushdownTest
Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.518 sec
Running org.apache.hadoop.hive.dynamodb.DynamoDBSerDeTest
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.031 sec

Results :

Tests run: 57, Failures: 0, Errors: 0, Skipped: 0

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running org.apache.hadoop.dynamodb.exportformat.ExportOutputFormatTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.036 sec
Running org.apache.hadoop.dynamodb.exportformat.ExportManifestEntryTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.04 sec
Running org.apache.hadoop.dynamodb.exportformat.ExportFileFlusherTest
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.354 sec

Results :

Tests run: 11, Failures: 0, Errors: 0, Skipped: 0

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.