Not picking up fs.s3a.stsAssumeRole.arn property

[vatsalvora]

We are trying to use the Hadoop config to assume a role when writing/reading to s3. The library is not assuming the role when reading/writing to s3.

[rumeshkrish]

@vatsalvora Can you please provide more details on the use case. ? Alternatively I hope you can enable Glue Service Role with tagging permission to use this library.

[mgerlach]

Likewise, fs.s3.customAWSCredentialsProvider (Spark configuration) and fs.s3.authorization.roleMapping (EMR cluster emrfs-site configuration) are not being picked up.

Context:

• Glue 4.0, Glue service role has sts:AssumeRole permission. With format("parquet") and the property fs.s3.customAWSCredentialsProvider set to the class name of an AWSCredentialsProvider implementation on the classpath that uses sts:AssumeRole to assume a role with s3:* permissions in a different account, we can write to S3 buckets in that account.
• EMR 6.11.1, EMR exec role has sts:AssumeRole. With format("parquet") and the property fs.s3.authorization.roleMapping set such that EMR would assume a role with s3:* permissions in a different account for certain buckets/prefixes, we can write to S3 buckets in that account.

Now, when using format("s3.parquet"), we get "access denied" errors and no files are written to the other account.

Like this, writing and tagging work only when writing to local (same account) S3 buckets.

(Let me know if you prefer (a) separate issue(s) for this.)