I want to register an Iceberg catalog of type AWS Glue in Flink with the code like this:
var properties = new HashMap<>();
properties.put("type", "iceberg");
properties.put("catalog-type", "glue");
properties.put("catalog-impl", "org.apache.iceberg.aws.glue.GlueCatalog");
properties.put("warehouse", "s3://my-bucket/path/to/warehouse");
var factory = new FlinkCatalogFactory();
var catalog = factory.createCatalog("glue_catalog", properties);
I develop my application in a local Kubernetes cluster, so I cannot use the following available authentication options:
The default credential provider (there is no profile file in the container).
Have the client assume an AWS role (there are no EC2 credentials available).
Instead, I want to use static client credentials (a pair of access key ID and secret). I couldn't find the user-facing documentation on how do do that, so I resorted to reading the source code.
Would a PR be accepted that in addition to the s3.* parameters mentioned above added the support for similarly named aws.* parameters? Such parameters would consistently apply to all AWS clients instantiated by DefaultAwsClientFactoryand could eventually deprecate the s3.* ones..
Feature Request / Improvement
I want to register an Iceberg catalog of type AWS Glue in Flink with the code like this:
I develop my application in a local Kubernetes cluster, so I cannot use the following available authentication options:
Instead, I want to use static client credentials (a pair of access key ID and secret). I couldn't find the user-facing documentation on how do do that, so I resorted to reading the source code.
Currently, static client credentials can be only configured for the S3 client: https://github.com/apache/iceberg/blob/0e7aa84b1dd378b4be56f5b45b6744b383501bd9/aws/src/main/java/org/apache/iceberg/aws/s3/S3FileIOProperties.java#L192-L216
These credentials do not affect the other clients, e.g. the AWS Glue one. As a result, in order to configure static credentials for the entire Glue catalog, one needs to implement a custom credentials provider using these parameters: https://github.com/apache/iceberg/blob/c68abfc9fd3956077b43aba20441f089bb8b93d6/aws/src/main/java/org/apache/iceberg/aws/AwsClientProperties.java#L37-L60
For example: https://github.com/apache/iceberg/blob/7071dc18ed66454542f466b5bfe8821028f2db0c/aws/src/test/java/org/apache/iceberg/aws/TestAwsClientFactories.java#L243-L253
Would a PR be accepted that in addition to the
s3.*
parameters mentioned above added the support for similarly namedaws.*
parameters? Such parameters would consistently apply to all AWS clients instantiated byDefaultAwsClientFactory
and could eventually deprecate thes3.*
ones..Query engine
None