search

googleapis / google-auth-library-java

Open source Auth client library for Java
https://developers.google.com/identity
BSD 3-Clause "New" or "Revised" License
405 stars 218 forks source link

Workload identity federation doesn't support full aws credential sources. #1408

Open ksauzz opened 1 month ago

ksauzz commented 1 month ago

InternalAwsSecurityCredentialsSupplier only support environment variables or EC2 metadata server to get AWS credential.

In my usecase, I can't use workload identity federation from AWS Glue (spark) to load data to BigQuery table using spark-bigquery-connector. This spark environment has no EC2 metadata endpoint, and spark driver process' environment variables cannot be updated from a job.

Environment details

AWS Glue 4.0 (spark) + pyspark

Steps to reproduce

  1. Prepare workload identity federation settings
  2. run AWS Glue job

External references such as API reference guides

Any additional information below

I think AWS SDKs including aws-sdk-java provide comprehensive ways to get credential from various AWS environments, so it would be nice to use DefaultCredentialsProvider or something instead of custom implementation in this library. But I guess google team wouldn't like to use such other vendor library...

DefaultCredentialsProvider's docs

AWS credentials provider chain that looks for credentials in this order:

  1. Java System Properties - aws.accessKeyId and aws.secretKey
  2. Environment Variables - AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY
  3. Credential profiles file at the default location (~/.aws/credentials) shared by all AWS SDKs and the AWS CLI
  4. Credentials delivered through the Amazon EC2 container service if AWS_CONTAINER_CREDENTIALS_RELATIVE_URI" environment variable is set and security manager has permission to access the variable,
  5. Instance profile credentials delivered through the Amazon EC2 metadata service
lsirac commented 1 month ago

Hi @ksauzz, you can supply your own custom AWS credential supplier to the library that handles your use case. See here.

ksauzz commented 1 month ago

I think It doesn't work for spark-bigquery-connector because the connector doesn't the config item to change the supplier. I hope core auth library would have this functionality without any patches by users. Otherwise, GCP users have to make a patch to each google libraries involving google-auth-library-java. Thank you.