Closed istreeter closed 4 weeks ago
Delta has a "S3 multi-cluster" mode, which allows safe cocurrent writes by multiple loaders.
To enable this mode, users can set the spark options documented by Delta:
spark.delta.logStore.s3.impl=io.delta.storage.S3DynamoDBLogStore
spark.io.delta.storage.S3DynamoDBLogStore.ddb.tableName=???
spark.io.delta.storage.S3DynamoDBLogStore.ddb.region=???
This commit adds the necessary runtime dependencies for this mode.
Delta has a "S3 multi-cluster" mode, which allows safe cocurrent writes by multiple loaders.
To enable this mode, users can set the spark options documented by Delta:
spark.delta.logStore.s3.impl=io.delta.storage.S3DynamoDBLogStorespark.io.delta.storage.S3DynamoDBLogStore.ddb.tableName=???spark.io.delta.storage.S3DynamoDBLogStore.ddb.region=???This commit adds the necessary runtime dependencies for this mode.