search

datafusion-contrib / datafusion-objectstore-s3

S3 as an ObjectStore for DataFusion
Apache License 2.0
59 stars 13 forks source link

Support aws_config::load_from_env() #5

Closed matthewmturner closed 2 years ago

matthewmturner commented 2 years ago

As @seddonm1 notes, aws_config::load_from_env() doesnt allow specifying endpoint which is required for integration testing.

that being said, it would still be nice to support this feature.

Maybe we could add a boolean argument to AmazonS3FileSystem::new like load_from_env and dispatch accordingly.

seddonm1 commented 2 years ago

Im not 100% sure but i think if you pass all None values to new_client the default credentials provider will do the env vars. I don't have an S3 bucket to test with though.

matthewmturner commented 2 years ago

I believe i tested that before and had same issue, but i will give it a try again.

matthewmturner commented 2 years ago

Yeah i get Error: Internal("ConstructionFailure(MissingCredentials)") when everything is set to None except bucket in AmazonS3FileSystem::new