Open d-v-b opened 3 years ago
actually it looks like the key change was in this PR: https://github.com/zarr-developers/zarr-python/pull/718
My best guess at the moment is that normalization is being called multiple times. I've certainly noticed the need to carefully set normalize_keys
on the FSStore contructor elsewhere, but hadn't yet tracked down what was going on.
cc: @martindurant
@d-v-b : is this still an issue?
I transferred this issue from zarr-python
to n5py
.
N5 requires that the dimension ordering of chunk keys be reversed. I.e., for a chunk key
foo/bar/0.1.2
, invert_chunk_coords will returnfoo/bar/2.1.0
.Pre-2.81, chunk keys used two delimiters: "/" delimits the hierarchy, and "." delimits the chunk indices. After 2.8.1, chunk keys can take the form
foo/bar/0/1/2
, and suddenly there is no distinction between hierarchy and chunk index delimiters. "/"-separated chunk keys can arise if the array's chunk store has akey_separator
property:Array._chunk_key()
.N5Store
has adimension_separator
property but not akey_separator
property, so forN5Store
chunk keys will use the default "." separator, andinvert_chunk_coords
will work fine.But
FSStore
uses thekey_separator
property for determining how to write keys to storage. Arrays created usingFSStore
will read thekey_separator
property of the store adapt chunk keys accordingly. Thus, attempts to extendFSStore
to work for N5 (wherekey_separator
is "/") will run into issues where chunk keys look likefoo/bar/0/1/2
and it becomes impossible forinvert_chunk_coords
to parse this into (array_prefix
,chunk_coordinate
)Zooming out, it seems like a leaky abstraction if the format of chunk keys changes as a function of the implementation details of the storage layer. Chunk keys should have a consistent, parseable format, and it is the job of the store to convert chunk keys into the names of objects in storage.