Closed GrigorievNick closed 2 years ago
As fast and dirty workaround, user can specify credentials through spark env variable pass through mechanic on YARN. \ Or specify env variables on container level in MESOS and K8s clusters. \ Spark conf for pass though on YARN.
spark.yarn.appMasterEnv.AWS_SECRET_ACCESS_KEY | AWS_ACCESS_KEY_ID
spark.executorEnv.AWS_SECRET_ACCESS_KEY | AWS_ACCESS_KEY_ID
But because delta rely on spark/hadoop-aws dependency to work with s3. I believe delta provider must support more canonical
way to configure AWS credentials.
Bug
Describe the problem
When configure spark using setup-configuration-s3-multi-cluster In example s3a credentials have configure with
hadoop.fs.s3a.*
properties. \ But there is another option in documentationspark.io.delta.storage.S3DynamoDBLogStore.credentials.provider
. where used DefaultAWSCredentialsProviderChainby default. \ And because of code implementation there is no way to specify
org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider` which take credentials from hadoop conf. \ Because delta create S3DynamoDBLogStore create AWS credentials provider using reflection with empty constructor.Steps to reproduce
Write any spark application which write new data to delta table. \ And configure it to use
S3DynamoDBLogStore
using delta.io official documentation.VERY IMPORTANT NOTES - Spark Cluster must not have valid AWS credentials on nodes.
.aws/credentials
--conf
Observed results
Application will fail with access denied.
Expected results
Delta commit happen correctly and dynamoDB table with locks updated.
Further details
Full stacktrace
Environment information
Willingness to contribute
The Delta Lake Community encourages bug fix contributions. Would you or another member of your organization be willing to contribute a fix for this bug to the Delta Lake code base?