alamb
commented
1 month ago BTW here is an example of how we use the AWS SKD to get credentials in datafusion-cli:
Open tustvold opened 1 month ago
alamb
commented
1 month ago BTW here is an example of how we use the AWS SKD to get credentials in datafusion-cli:
Description
Problem
object_store provides a limited selection of common authentication mechanisms, with a particular focus on those used in server contexts. It does not, however, seek to replicate the entire credentials setup of the various SDKs. Polars is often run on end user devices and therefore people wish for it to support a broader range of authentication options.
object_store exposes a CredentialProvider API that can be used to provide an alternative way to source credentials.
Proposal
https://github.com/pola-rs/polars/issues/18979 tracks exposing CredentialProvider in a way that it can be configured, there are, however, some design questions around what this might look like through a python API.
An alternative would be for polars to provide an option to use aws-sdk-rust to source credentials, much like datafusion-cli does.
Alternatives Considered
Users could use software like aws-vault to generate session credentials, whilst this has other security benefits, for various reasons people may not wish to do this.
We could expose the full CredentialProvider API to users. This would be more flexible, support providers other than AWS, and avoid adding some non-trivial additional dependencies, but requires more design work.
Related Context