earth-mover / icechunk

Open-source, cloud-native transactional tensor storage engine
https://icechunk.io
Apache License 2.0
290 stars 17 forks source link

Support Google Cloud Storage for the object store #265

Open paraseba opened 1 month ago

geospatial-jeff commented 4 weeks ago

GCP buckets may be configured to support the S3 API. Technically speaking icechunk is already capable of accessing data in GCS. It just requires some setup by the bucket owner to configure S3 API compatibility - https://cloud.google.com/storage/docs/interoperability

Personally I'm a big fan of only supporting S3 and relying on cloud providers for interoperability. It really sucks to clutter a library with a bunch of slightly different blob store implementations to deal with the slightly different APIs exposed by various cloud vendors. I don't think the complexity is worth it, it comes with a cost.