[Bug]: Extension namespaces not exported from HDF5 NWB

[sneakers-the-rat]

What happened?

Testing an export of a dataset that has a custom namespace/schema, and found that it isn't exported and so a subsequent read errors.

Taking a look at the __cache_spec method, it seems like the writer uses its own manager rather than the source file's manager: https://github.com/hdmf-dev/hdmf-zarr/blob/5e48ab5e04512c5f5151d0d372eb2284f85dfeaa/src/hdmf_zarr/backend.py#L284

which is confirmed in a debugger. Setting a breakpoint at the start of that function:

(Pdb) >? self.manager
<hdmf.build.manager.BuildManager object at 0x12cd55490>
(Pdb) >? self.manager.namespace_catalog.namespaces
('hdmf-common', 'hdmf-experimental', 'core')

and then going up to the initial call which has the source IO object:

(Pdb) >? read_io.manager
<hdmf.build.manager.BuildManager object at 0x11029d610>
(Pdb) >? read_io.manager.namespace_catalog.namespaces
('hdmf-common', 'hdmf-experimental', 'core', 'ndx-aibs-ecephys')

If I do this just before calling __cache_spec:

(Pdb) >? self.manager._BuildManager__type_map._TypeMap__ns_catalog = src_io.manager.namespace_catalog

then the namespace is correctly written, but the read still errors. The namespace is listed in namespaces here: https://github.com/hdmf-dev/hdmf-zarr/blob/5e48ab5e04512c5f5151d0d372eb2284f85dfeaa/src/hdmf_zarr/backend.py#L207

but when it reaches BuildManager.load_namespaces, the deps are empty. That's as far as i'm gonna make it tonight.

Steps to Reproduce

from pynwb import NWBHDF5IO
from hdmf_zarr.nwb import NWBZarrIO

in_file = "/location/of/data/nwb/sub-738651046_ses-760693773.nwb"
out_file = "./test.zarr"

with NWBHDF5IO(in_file, 'r', load_namespaces=True) as read_io:
    with NWBZarrIO(out_file, mode="w") as export_io:
        export_io.export(src_io=read_io, write_args={'link_data':False})

zr = NWBZarrIO(out_file, 'r', load_namespaces=True)
zf = zr.read()

Traceback

Traceback (most recent call last):
  File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/pydevconsole.py", line 364, in runcode
    coro = func()
           ^^^^^^
  File "<input>", line 2, in <module>
  File "/location/of/fork/hdmf-zarr/venv/lib/python3.11/site-packages/hdmf/utils.py", line 664, in func_call
    return func(args[0], **pargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/location/of/fork/hdmf-zarr/venv/lib/python3.11/site-packages/hdmf/backends/io.py", line 60, in read
    container = self.__manager.construct(f_builder)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/location/of/fork/hdmf-zarr/venv/lib/python3.11/site-packages/hdmf/utils.py", line 664, in func_call
    return func(args[0], **pargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/location/of/fork/hdmf-zarr/venv/lib/python3.11/site-packages/hdmf/build/manager.py", line 284, in construct
    result = self.__type_map.construct(builder, self, None)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/location/of/fork/hdmf-zarr/venv/lib/python3.11/site-packages/hdmf/utils.py", line 664, in func_call
    return func(args[0], **pargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/location/of/fork/hdmf-zarr/venv/lib/python3.11/site-packages/hdmf/build/manager.py", line 795, in construct
    return obj_mapper.construct(builder, build_manager, parent)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/location/of/fork/hdmf-zarr/venv/lib/python3.11/site-packages/hdmf/utils.py", line 664, in func_call
    return func(args[0], **pargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/location/of/fork/hdmf-zarr/venv/lib/python3.11/site-packages/hdmf/build/objectmapper.py", line 1228, in construct
    subspecs = self.__get_subspec_values(builder, self.spec, manager)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/location/of/fork/hdmf-zarr/venv/lib/python3.11/site-packages/hdmf/build/objectmapper.py", line 1157, in __get_subspec_values
    self.__get_sub_builders(groups, spec.groups, manager, ret)
  File "/location/of/fork/hdmf-zarr/venv/lib/python3.11/site-packages/hdmf/build/objectmapper.py", line 1208, in __get_sub_builders
    ret.update(self.__get_subspec_values(sub_builder, subspec, manager))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/location/of/fork/hdmf-zarr/venv/lib/python3.11/site-packages/hdmf/build/objectmapper.py", line 1157, in __get_subspec_values
    self.__get_sub_builders(groups, spec.groups, manager, ret)
  File "/location/of/fork/hdmf-zarr/venv/lib/python3.11/site-packages/hdmf/build/objectmapper.py", line 1200, in __get_sub_builders
    sub_builder = self.__flatten(sub_builder, subspec, manager)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/location/of/fork/hdmf-zarr/venv/lib/python3.11/site-packages/hdmf/build/objectmapper.py", line 1213, in __flatten
    tmp = [manager.construct(b) for b in sub_builder]
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/location/of/fork/hdmf-zarr/venv/lib/python3.11/site-packages/hdmf/build/objectmapper.py", line 1213, in <listcomp>
    tmp = [manager.construct(b) for b in sub_builder]
           ^^^^^^^^^^^^^^^^^^^^
  File "/location/of/fork/hdmf-zarr/venv/lib/python3.11/site-packages/hdmf/utils.py", line 664, in func_call
    return func(args[0], **pargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/location/of/fork/hdmf-zarr/venv/lib/python3.11/site-packages/hdmf/build/manager.py", line 280, in construct
    result = self.__type_map.construct(builder, self, parent)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/location/of/fork/hdmf-zarr/venv/lib/python3.11/site-packages/hdmf/utils.py", line 664, in func_call
    return func(args[0], **pargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/location/of/fork/hdmf-zarr/venv/lib/python3.11/site-packages/hdmf/build/manager.py", line 795, in construct
    return obj_mapper.construct(builder, build_manager, parent)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/location/of/fork/hdmf-zarr/venv/lib/python3.11/site-packages/hdmf/utils.py", line 664, in func_call
    return func(args[0], **pargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/location/of/fork/hdmf-zarr/venv/lib/python3.11/site-packages/hdmf/build/objectmapper.py", line 1228, in construct
    subspecs = self.__get_subspec_values(builder, self.spec, manager)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/location/of/fork/hdmf-zarr/venv/lib/python3.11/site-packages/hdmf/build/objectmapper.py", line 1157, in __get_subspec_values
    self.__get_sub_builders(groups, spec.groups, manager, ret)
  File "/location/of/fork/hdmf-zarr/venv/lib/python3.11/site-packages/hdmf/build/objectmapper.py", line 1186, in __get_sub_builders
    for parent_dt in manager.namespace_catalog.get_hierarchy(ns, dt):
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/location/of/fork/hdmf-zarr/venv/lib/python3.11/site-packages/hdmf/utils.py", line 664, in func_call
    return func(args[0], **pargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/location/of/fork/hdmf-zarr/venv/lib/python3.11/site-packages/hdmf/spec/namespace.py", line 328, in get_hierarchy
    raise KeyError("'%s' not a namespace" % namespace)
KeyError: "'ndx-aibs-ecephys' not a namespace"

Operating System

macOS

Python Executable

Python

Python Version

3.11

Package Versions

environment_for_issue.txt

Code of Conduct

• [X] I agree to follow this project's Code of Conduct
• [X] Have you checked the Contributing document?
• [X] Have you ensured this bug was not already reported?

[oruebel]

Thanks for the issue, we'll take a look.

[oruebel]

@mavaylon1

[mavaylon1]

I'll tackle this.

[bjhardcastle]

Importing the packages corresponding to the custom namespaces seems to allow reading to work, if anyone needs a temporary fix.

[mavaylon1]

Importing the packages corresponding to the custom namespaces seems to allow reading to work, if anyone needs a temporary fix.

Thanks for the note!