search

zarr-developers / zarr-specs

Zarr core protocol for storage and retrieval of N-dimensional typed arrays
https://zarr-specs.readthedocs.io/
Creative Commons Attribution 4.0 International
88 stars 28 forks source link

Non-deletable stores. #103

Open Carreau opened 4 years ago

Carreau commented 4 years ago

I just realized that the zipStore does not allow deletion (and thus does not allow rename). Is that something that should be addressed in the spec, of should it maybe be fixed in the ZipStore with TombStones ?

alimanfoo commented 4 years ago

Hi @Carreau, I think it is reasonable if a store does not support deletion. If you agree, then yes I'd suggest we clarify that supporting deletion is optional somehow. E.g., maybe stores can be readable, writable, deletable, listable, these are the basic capabilities. A store that is only readable can still be useful. Similarly a store that is only readable and writable can still be useful.

Carreau commented 3 years ago

That's fine with me, but that's push more complexity on the Zarr Protocol implementation, while you could try to push the deletion implementation details to the store. That seem also to be there only to accommodate the zip store.

jstriebel commented 1 year ago

The spec currently states:

A listable store supports any one or more of the following operations:

So this includes that there may be stores that do not support deletion (such as any read-only store would, too). I assume that resolves this issue, please re-open if that's not the case.

jakirkham commented 1 year ago

Think what Matthias was referring to is ZipStore allows writing, but not deletion. So the erase* functions don't work. Similarly writing behaves more like write-once (since deletion is not supported).

That said, think in practice ZipStore behaves better as a "readable store" (ignoring the initial creation step). This is already a common case like a read-only filesystem or HTTP stores is a common use case that we would want to support, which is covered under "readable store".

Side note: We may prefer to fix the ZipStore oddity ( https://github.com/zarr-developers/zarr-python/issues/828 ) either by making it more feature complete ( https://github.com/zarr-developers/zarr-python/pull/1184 ) or moving it to read-only.

jstriebel commented 1 year ago

Thanks for the clarification @jakirkham! Do you think there's a TODO for the current v3 core spec from this issue? I'm re-opening it, but I'm not quite sure if this belongs into the zarr-specs repo or rather zarr-python? Sorry for the confusion, seems like I'm missing some context here. @joshmoore, can you help?

joshmoore commented 1 year ago

Seems like there's enough confusion around this that we should minimally try to add our expectations in the V3 spec and possibly V2. @Caviere will soon be investigating the ZipStore and we can try to dig into these questions which may then lead to wanting to address delete-ability more broadly in the spec.

jstriebel commented 1 year ago

Since the zip store is currently not yet specified in v3 I assume this does not block ZEP 1, which is mostly about the core spec. Further clarifications can also be added to the core spec after ZEP 1 is accepted.