Consolidate Metadata

[mavaylon1]

Motivation

What was the reasoning behind this change? Please explain the changes briefly.

Addressing #139. By default, the metadata will be consolidated into one file, .zmetadata. This will check the existence of this file and will still open the file even if the metadata is not consolidated.

• [x] Boolean for consolidating metadata
• [x] Build tests for when metadata is and is not consolidated
• [x] General zarr file builder for tests
• [x] Update docs with notes on metadata updates.

How to test the behavior?

Show how to reproduce the new behavior (can be a bug fix or a new feature)

Checklist

• [x] Did you update CHANGELOG.md with your changes?
• [x] Have you checked our Contributing document?
• [x] Have you ensured the PR clearly describes the problem and the solution?
• [x] Is your contribution compliant with our coding style? This can be checked running ruff from the source directory.
• [x] Have you checked to ensure that there aren't other open Pull Requests for the same change?
• [x] Have you included the relevant issue number using "Fix #XXX" notation where XXX is the issue number? By including "Fix #XXX" you allow GitHub to close issue #XXX when the PR is merged.

[mavaylon1]

Fix #139

[codecov-commenter]

Codecov Report

Attention: 2 lines in your changes are missing coverage. Please review.

Comparison is base (b51e127) 86.10% compared to head (f23e3f3) 86.45%. Report is 5 commits behind head on dev.

Files	Patch %	Lines
tests/unit/utils.py	71.42%	2 Missing :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## dev #142 +/- ## ========================================== + Coverage 86.10% 86.45% +0.35% ========================================== Files 13 14 +1 Lines 3189 3279 +90 ========================================== + Hits 2746 2835 +89 - Misses 443 444 +1 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[mavaylon1]

Comments for review

1. There are a few instances in backend.py that remain zarr.open() rather than __open_consolidated.

• In Line 81, I left this as a simple test of whether it can be read.
• In Line 199, the goal is to load the cached namespace from a file. The parameters in this zarr.open() don't include any of the optional arguments for __open_consolidated, and the whole point of having a consolidated metadata is for quick look-ups for groups, datasets, etc (nothing to do with the namespace). As a result, I think it might be "safer" (meaning less likely for a bug to affect this area) if we kept it as is. That being said it is an easy switch if the majority disagrees with this view.

2. Why are there only two tests? With the zarr.consolidate_metadata being called at the end of write_builder, it will generate test examples (including examples with references) that have zmetadata, meaning it's covered by existing tests. (As seen in coverage).

3. Why is there no consolidate_metadata method anymore? This method was here briefly in this PR, but I removed it since it is literally calling the same method within zarr. I think wrapping it with the same name would just create confusion if something goes wrong, meaning the issue is not us but zarr. I don't want us to have issue tickets saying "it's broken" just because they are not using the zarr method correctly.

[oruebel]

Thanks for the additional comments.

1. There are a few instances in backend.py that remain zarr.open() rather than __open_consolidated.

   • In Line 81, I left this as a simple test of whether it can be read.

   • In Line 199, the goal is to load the cached namespace from a file.

I agree, it makes sense to leave zarr.open in those two instances.

2. Why are there only two tests? With the zarr.consolidate_metadata being called at the end of write_builder, it will generate test examples (including examples with references) that have zmetadata, meaning it's covered by existing tests. (As seen in coverage).

Makes sense. This raises the question whether we need some flag in the tests to run them with and without consolidation of metadata. I.e., I assume now those tests always run with consolidated metadata but no longer test for the case where metadata is not consolidated.

3. Why is there no consolidate_metadata method anymore? This method was here briefly in this PR, but I removed it since it is literally calling the same method within zarr. I think wrapping it with the same name would just create confusion if something goes wrong, meaning the issue is not us but zarr. I don't want us to have issue tickets saying "it's broken" just because they are not using the zarr method correctly.

When would the user directly need to call consolidate_metadata? If there is a case where a user will need to call this method in zarr, then I think it would be useful to mention that somewhere in the docs.

[mavaylon1]

Thanks for the additional comments.

1. There are a few instances in backend.py that remain zarr.open() rather than __open_consolidated.

   • In Line 81, I left this as a simple test of whether it can be read.
   • In Line 199, the goal is to load the cached namespace from a file.

I agree, it makes sense to leave zarr.open in those two instances.

2. Why are there only two tests? With the zarr.consolidate_metadata being called at the end of write_builder, it will generate test examples (including examples with references) that have zmetadata, meaning it's covered by existing tests. (As seen in coverage).

Makes sense. This raises the question whether we need some flag in the tests to run them with and without consolidation of metadata. I.e., I assume now those tests always run with consolidated metadata but no longer test for the case where metadata is not consolidated.

3. Why is there no consolidate_metadata method anymore? This method was here briefly in this PR, but I removed it since it is literally calling the same method within zarr. I think wrapping it with the same name would just create confusion if something goes wrong, meaning the issue is not us but zarr. I don't want us to have issue tickets saying "it's broken" just because they are not using the zarr method correctly.

When would the user directly need to call consolidate_metadata? If there is a case where a user will need to call this method in zarr, then I think it would be useful to mention that somewhere in the docs.

2. Well with this PR, it will always be consolidated. I think it raises a question for how many zarr files are in the wild already (assuming with the Allen Institute) that are from hdmf-zarr prior to this PR. Even so, I can use example.zarr for a test for this case.
3. The user would need to call it again if they edit the file after it has been written. I can make a quick tutorial for sure.

[oruebel]

Can you also add a brief section on consolidation of metadata to https://github.com/hdmf-dev/hdmf-zarr/blob/dev/docs/source/storage.rst so that we have a record in the docs on where this is stored and why.

[oruebel]

2. Well with this PR, it will always be consolidated.

I don't think we want to make the hard assumption that metadata will always be consolidated.

[mavaylon1]

This looks overall good to me. One quick question, are we still running any tests now with consolidate_metadata=False or all tests now run with consolidated metadata only?

There's one test to check that it runs with False, in relation to consolidated metadata. However, the rest is using the default True. I don't think it's a big enough change to test for the entire test suite for both but thoughts?

[oruebel]

I don't think it's a big enough change to test for the entire test suite for both but thoughts?

I think that would be nice, if only because it would help safeguard against potential issues as we make changes in the future. That being said, configuring the test suite to run in both modes is probably a non-trivial undertaking and is not worth holding up this PR for. Please just create an issue for future reference and we can decide separately on how to proceed on that question.