https://github.com/hdmf-dev/hdmf-zarr/pull/184 adds a bug fix for reading files from S3 with consolidated metadata. However, we currently are not testing reading with consolidated metadata only for local files. Since consolidated metadata is particularly relevant for reading from S3, it would be useful to add tests to ensure this is working correctly with files on S3 as well.
Steps to Reproduce
I believe we currently do no have a test file with consolidated metadata on DANDI. It would be useful to add some small Zarr files with and without consolidated metadata to our test DANDI dataset so that we can test that the remote files are being opened correctly in the streaming tests. Currently we are using a dataset from the Allen Institute for the streaming tests but it would be useful to test the behavior for streaming access more thoroughly using our own files.
What happened?
https://github.com/hdmf-dev/hdmf-zarr/pull/184 adds a bug fix for reading files from S3 with consolidated metadata. However, we currently are not testing reading with consolidated metadata only for local files. Since consolidated metadata is particularly relevant for reading from S3, it would be useful to add tests to ensure this is working correctly with files on S3 as well.
Steps to Reproduce
I believe we currently do no have a test file with consolidated metadata on DANDI. It would be useful to add some small Zarr files with and without consolidated metadata to our test DANDI dataset so that we can test that the remote files are being opened correctly in the streaming tests. Currently we are using a dataset from the Allen Institute for the streaming tests but it would be useful to test the behavior for streaming access more thoroughly using our own files.
To test whether a file is indeed opened correctly with consolidated metadata we can use the following check in the unit tests : https://github.com/hdmf-dev/hdmf-zarr/blob/afaab5fbca03a0979d9f234e13e43027f545c366/tests/unit/test_zarrio.py#L160-L166
Traceback
Operating System
macOS
Python Executable
Conda
Python Version
3.9
Package Versions
No response
Code of Conduct