Support Amazon EKS Pod Identity

[desmondcheongzx]

Is your feature request related to a problem?

In https://dist-data.slack.com/archives/C041NA2RBFD/p1731789019058459 we have a user with a production cluster using pod identity associations instead of IAM roles for service accounts (IRSA). The default configuration for AWS does not pick up credentials for EKS pod identity associations.

Describe the solution you'd like

Add support for pod identity associations, which are a different credential provider which doesn’t use assume role with web identity. They were more recently introduced at the end of 2023 and are the current recommendation for IAM permissions.

Additional Context

• https://docs.aws.amazon.com/eks/latest/userguide/pod-identities.html
• https://aws.amazon.com/blogs/containers/amazon-eks-pod-identity-a-new-way-for-applications-on-eks-to-obtain-iam-credentials/

[jaychia]

Update: created a cluster with Pod Identity. Going to test boto3 and Daft on it now.

[jaychia]

Reproduced the issue!

I believe this is because our version of the AWS SDK is too old, and this is already fixed in newer versions of the SDK. I'd like to propose that we make a tracking issue for the upgrade of our SDK (which is fairly involved, given all the breaking changes that were made since we started building on it a few years ago).