earth-mover / icechunk

Open-source, cloud-native transactional tensor storage engine
https://icechunk.io
Apache License 2.0
305 stars 18 forks source link

Azure blob suppport #411

Open qianqianwang-microsoft opened 4 days ago

qianqianwang-microsoft commented 4 days ago

Adding support for azure blob storage using azure_storage and azure_storage_blobs crates. This PR is still a WIP.

kylebarron commented 4 days ago

My own opinion is that the maintenance burden would be lower if icechunk used one API on top of the object_store crate rather than reimplementing object store operations individually for S3, GCS, and Azure.

rabernat commented 4 days ago

@qianqianwang-microsoft - thanks so much for this PR! We are really excited to work with you to support Azure in Icechunk!

@kylebarron thanks for chiming in. I generally agree with you. We had some good discussion of this in https://github.com/earth-mover/icechunk/issues/266#issuecomment-2455881019. But that discussion stalled out (and @paraseba is OoO this week.)

We are very open to using an object_store-based approach in Icechunk, provided that: