[Feature]: Always consolidate store structure

[CodyCBakerPhD]

What would you like to see added to HDMF-ZARR?

From discussion on https://github.com/dandi/dandi-archive/issues/1745#issuecomment-1806440234

Zarr command: https://zarr.readthedocs.io/en/stable/api/convenience.html#zarr.convenience.consolidate_metadata

Always enable consolidation of metadata on NWB file creation to make it easier to traverse store contents and on S3, reduce the number of head-like requests that would be needed to read store structure

Is your feature request related to a problem?

No response

What solution would you like?

Wish the NWBZarrIO or higher level ZarrIO would do this automatically on file creation, but unsure where in the stack to inject that

Do you have any interest in helping implement the feature?

Yes.

Code of Conduct

• [X] I agree to follow this project's Code of Conduct
• [X] Have you checked the Contributing document?
• [X] Have you ensured this change was not already requested?

[oruebel]

Interesting, I wasn't aware of this feature. The call to create the consolidated metadata I think would need to happen at the end of 'ZarrIO.write_builder' and then 'ZarrIO.open' wound need to check if the consolidated metadata exist to use 'open_consoludated' instead. We'd probably want to also add a 'ZarrIO.consolidate_metadata(..)' function to explicitly update/generate the metadata file and add a flag on write to allow to enable/disable on write.

[oruebel]

@mavaylon1 do you think you can take a stab at this?

[mavaylon1]

@mavaylon1 do you think you can take a stab at this?

I shall.