Feature request: allow user to use a boto3-native credential provider/session for s3 etc.

jwhitaker-gridcog commented 5 months ago

Description

It's nice that Polars has native support for s3, however we have issues with using it with less common credential setups.

In short, I would really like it it I could just rely on boto3 to manage credentials for me - there's a long tail of things that the rust-side object_store seems to struggle with. (for context, when I talk about polars i mean py-polars).

Examples I have hit personally:

passing profile_name in storage_options (workaround: open a boto client and export with get_frozen_credentials)
handling assumed roles (workaround: open a boto client, assume the role, export the sts response; or, use an approach like https://github.com/boto/boto3/pull/3253/files to make the boto client assume to role for you + handle refreshing etc).
you have probably hit more - I guess "making aws creds work like users expect from boto" is a recurring thing that pops up.

In particular, the workarounds are a problem when credentials expire. A boto session/credential handles refreshing for you. However if I'm passing a session token to polars myself via the above hacks, then I have to handle credential expiration myself, which I can't really do in a foolproof way - what happens if I have a long-lived lazyframe loaded with a 15-minute aws session, for example?

If I could just set up a boto3 session myself, and then pass it to polars and say "hey use this", this would be great from a UX perspective.

This could be general enough to go into a how to read from s3 101 doc, which could avoid the traps new users hit ("scan_parquet('s3://...') doesn't work, oh what's this, i have to go setting session tokens myself? ugh").

From your perspective, I wonder if it could simplify things a lot as well - instead of needing to provide an endless set of hooks to configure the rust side "kind of like a boto session", you could literally rely on boto itself.

On the engineering side though, this would require some glue from object_store back up to the python layer when it needs to fetch/refresh credentials - I don't know if this could be done cleanly/acceptably.