search

developmentseed / obstore

Simple, fast integration with Amazon S3, Google Cloud Storage, Azure Storage, and S3-compliant APIs like Cloudflare R2
https://developmentseed.org/obstore
MIT License
146 stars 3 forks source link

Feature request: support exclusive put mode #73

Closed martindurant closed 2 weeks ago

martindurant commented 2 weeks ago

This is the "conditional upload" feature, supported by AWS and others for atomic writes, equivalent to "xb" locally. The mode is important, for example, to icechunk, which uses the store atomic operation to ensure there is no race in setting the latest version of a dataset.

cf https://github.com/fsspec/filesystem_spec/issues/1693

martindurant commented 2 weeks ago

On closer reading, I suppose "create" does exactly this - apologies.

kylebarron commented 2 weeks ago

I think so, yes. We should test that it is indeed working on s3

kylebarron commented 2 weeks ago

See also https://github.com/apache/arrow-rs/issues/6285

martindurant commented 2 weeks ago

We should test that it is indeed working on s3

I bet it doesn't on moto... I hear that minio have had it a while, though.

kylebarron commented 2 weeks ago

I mean we should manually test this on an actual S3 bucket.