search

apache / arrow-rs

Official Rust implementation of Apache Arrow
https://arrow.apache.org/
Apache License 2.0
2.63k stars 803 forks source link

[object_storage] Add support for API-compatible S3 providers #6664

Closed rdelfin closed 3 weeks ago

rdelfin commented 3 weeks ago

Is your feature request related to a problem or challenge? Please describe what you are trying to do. Currently, if you want to work with a storage provider that provides the same API layer as S3, there's no way I could find to get the S3 provider to work for this purpose. However, the aws-sdk-s3 crate does support this (or at least, doesn't actively prevent you from using it that way).

Describe the solution you'd like This crate should not fail if the URL provided does not match an S3 URL schema, as long as all other relevant information is being provided.

Describe alternatives you've considered This could also be implemented as a separate provider, but I don't think the extra effort makes sense. Similarly, I could drop down and use the aws-sdk-s3 crate directly, but I do think object_store would benefit from supporting this feature, as I believe using an S3-like API is quite common for other storage solutions.

Additional context N/A

rdelfin commented 3 weeks ago

I see I missunderstood the difference between endpoint and url. Closing