Open hutch3232 opened 4 days ago
I doubt Polars has control over object_store
feature additions. I suggest you raise this request in their repo.
Oh, I somehow didn't realize they were separate libraries. Looks like it used to be experimentally supported but that support was dropped. Bummer.
https://github.com/apache/arrow-rs/pull/4238 https://github.com/apache/arrow-rs/issues/4556
Description
I have a variety of different AWS/S3 profiles in my
~/.aws/credentials
and~/.aws/config
files. I'd like to be able to either explicitly passprofile
intostorage_options
or implicitly by setting anAWS_PROFILE
environmental variable so that I can be sure to use the appropriate bucket keys/endpoint/and other configs.I saw here that profile is not listed as a supported option: https://docs.rs/object_store/latest/object_store/aws/enum.AmazonS3ConfigKey.html
polars
seems to use the first profile listed in those~/.aws
files, even if the profile name is not 'default'. By ensuring the relevant profile was listed first,pl.read_parquet("s3://my-bucket/my-parquet/*.parquet")
would work, but being order-dependent is confusing and not scalable.FWIW this functionality exists in
pandas
and I'm hoping to migrate code topolars
, but this is kind of essential.