Eventual-Inc / Daft

Distributed DataFrame for Python designed for the cloud, powered by Rust
https://getdaft.io
Apache License 2.0
1.76k stars 105 forks source link

[FEAT] I/O Support for Amazon S3 Express One Zone #2227

Open peter-gy opened 2 weeks ago

peter-gy commented 2 weeks ago

Is your feature request related to a problem? Please describe.

Currently it is not possible to read or write contents of an Amazon S3 Express One Zone directory bucket. The standard daft.read_* operations raise FileNotFoundError.

Describe the solution you'd like

I/O operations should work the same way they do with files stored in regular S3 buckets.

Additional context

As per @samster25, it is fair to assume that S3 Express handles credentials differently, therefore the forbidden 403 error gets propagated to FileNotFoundError.

Related Slack thread: https://dist-data.slack.com/archives/C052CA6Q9N1/p1714741124984589