FSStore doesn't see array in an https-backed group

[d-v-b]

It seems that FSStore can access the following group, but does not find the array stored in the group. The array can be accessed directly, however.

import zarr
url = 'https://uk1s3.embassy.ebi.ac.uk/idr/zarr/v0.4/idr0062A/6001240.zarr/'
group = zarr.open(zarr.storage.FSStore(url))
assert len(group.items()) == 0
array = zarr.open(zarr.storage.FSStore(url), path='0')
assert array.shape == (2, 236, 275, 271)

cc @martindurant

[martindurant]

Since HTTP can't really do file listing (except a few special cases derived from FTP), it can only be used with datasets that have consolidated metadata. Without [consolidated] metadata, zarr needs listing to know what arrays are contained in a group.

[jhamman]

@d-v-b - does this store have a consolidated metadata object (.zmetadata) at its root? Without it, it won't be able to list members.

edit: @martindurant beat me by 10 seconds. So fast!

[d-v-b]

yall are super fast! good to know re: consolidated metadata. This is a test dataset for ome-ngff, and the spec doesn't mention consolidated metadata, but it sounds like it should (as long as we want to use test data on http :) )

[joshmoore]

I'm very hesitant to mention consolidated metadata in OME-NGFF due to the lack of support in other Zarr implementations. I'm equally thrilled about having a ZEP for consolidated metadata in V3 so we can do exactly that!

[martindurant]

If you don't want to make a .zmetadata in the same location, you can put a kerchunk index anywhere else for the same effect :)

[joshmoore]

Same problem, same excitement.

[d-v-b]

I'm closing this, since the behavior I described is expected from the http store.