Add support for AWS IAM Profile Credentials for S3 variant

[morazow]

Situation

With Exasol Spark Connector (SEC) S3 variant we are accessing S3 buckets temporary data storage. To access the buckets users should provide AWS credentials (Access Key and Secret Key) with enough permission as Spark options.

However, spark-connector should also support using AWS instance profile credentials for the cases when Spark clusters is running in EC2 instances. This allows ease of use and less operational hurdles.

We add AWS DefaultCredentialsProviderChain to the underlying S3 client so that it looks for credentials in order:

• Static Credentials Provider (currently implemented) using user provided credentials
• Instance Profile Credentials Provider

Acceptance Criteria

• Validation errors are not thrown when user accessKey and secretKey are not provided
• Instance Profile Credentials are used when user credentials are not provided
• Validation errors with clear description provided if no credentials are available or valid

References

• AWS SDK Default Credentials Provider
• AWS SDK Instance Profile Credentials Provider

[morazow]

One possible limitation:

We should check first that Exasol CSV Loader can work with IAM Policy provided credentials.

[morazow]

IAM crendentials could be enable for CSV loader with database parameters: https://exasol.my.site.com/s/article/Changelog-content-15155?language=en_US.

I would suggest to extend the effort since this would also require changes in the CSV query generator classes.

[Shmuma]

7 days