search

zarr-developers / community

An open community with an interest in developing and using new technologies for tensor data storage.
19 stars 1 forks source link

Cloud Storage #13

Open jakirkham opened 6 years ago

jakirkham commented 6 years ago

There has been a lot of working on getting Zarr/N5 to use cloud storage. Some references here (and many more that I'm sure I missed). ( https://github.com/zarr-developers/zarr/pull/252 ) ( https://github.com/zarr-developers/zarr/pull/288 ) ( docs ) ( n5-google-cloud ) ( n5-aws-s3 ) Would be good if we could formulate a bit of a strategy in terms of how we provide cloud storage and how we can plug that into the various language implementations that Zarr/N5 have been growing.

martindurant commented 6 years ago

I have, of course, been a proponent of filesystem standardisation, and I think that is also material here. We are probably not in any position to do so, but such a standard could be cross-language, and so it would be enough to have some backend implement it for zarr to be able to use it. There has not been much interest in doing this, though...