search

projectnessie / nessie

Nessie: Transactional Catalog for Data Lakes with Git-like semantics
https://projectnessie.org
Apache License 2.0
909 stars 119 forks source link

Catalog S3: use DefaultCredentialsProvider if no access key configured #8987

Open adutra opened 5 days ago

adutra commented 5 days ago

Using DefaultCredentialsProvider as a last resort allows leveraging instance profile credentials delivered through the Amazon EC2 metadata service.

dorsegal commented 5 days ago

Will this work for eks IRSA/Pod Idendity?

adutra commented 5 days ago

Will this work for eks IRSA/Pod Idendity?

@dorsegal I can't tell for sure I'm afraid, but it seems so: IRSA uses WebIdentityTokenFileCredentialsProvider according to this SO question:

https://stackoverflow.com/questions/70964807/aws-java-sdk-v2-not-using-irsa-iam-role-in-eks

And DefaultCredentialsProvider uses WebIdentityTokenFileCredentialsProvider.

Do you think you could try this PR out in your environment?

adutra commented 5 days ago

Side note: GKE clusters have a similar way of binding roles to service accounts. Whatever solution we come up with here needs to be applied to GCS options as well. (and maybe to AKS / ADLS as well).

dorsegal commented 4 days ago

Will this work for eks IRSA/Pod Idendity?

@dorsegal I can't tell for sure I'm afraid, but it seems so: IRSA uses WebIdentityTokenFileCredentialsProvider according to this SO question:

https://stackoverflow.com/questions/70964807/aws-java-sdk-v2-not-using-irsa-iam-role-in-eks

And DefaultCredentialsProvider uses WebIdentityTokenFileCredentialsProvider.

Do you think you could try this PR out in your environment?

@adutra I tested it with pod identity and it works. IRSA has the same mechanism so it will work with it as well.

olivier-derom commented 3 days ago

Tested with IRSA and can confirm it works as well

snazy commented 2 days ago

Actually, we could introduce a similar mechanism like auth-type (GcsBucketOptions.GcsAuthType) to S3 and users can configure it explicitly to WORKLOAD_IDENTITY. WDYT?

olivier-derom commented 2 days ago

So instead of making it a fallback option, making it an option in clientAuthenticationMode? I like that idea as it will more clearly state the possibilities.