[Bug]: "ValueError: No specification for" on creating a new extension

What happened?

I am trying to follow the tutorial on creating new NWB extensions here however I am stuck after auto-generating the spec and creating the python module. When I try to import the module, the load_namespaces function fails to load the new type that I defined in create_extension_spec and that generated the two yamls, which look exactly like all yamls I could see from all extensions.

The extension I am trying to build is at: https://github.com/nwb-zebrafish/ndx-zebrafish the current state of the main branch

Steps to Reproduce

# Creating an editable install of the new extension linked above and trying 
from ndx_zebrafish import ZebrafishBehavior
# causes the traceback.

Traceback

Traceback (most recent call last):
  File "/home/vilimstih/miniconda3/envs/stytranwb/lib/python3.9/unittest/loader.py", line 34, in testFailure
    raise self._exception
ImportError: Failed to import test module: test_zebrafishbehavior
Traceback (most recent call last):
  File "/home/vilimstih/miniconda3/envs/stytranwb/lib/python3.9/unittest/loader.py", line 436, in _find_test_path
    module = self._get_module_from_name(name)
  File "/home/vilimstih/miniconda3/envs/stytranwb/lib/python3.9/unittest/loader.py", line 377, in _get_module_from_name
    __import__(name)
  File "/home/vilimstih/phd/ndx-zebrafish/src/pynwb/tests/test_zebrafishbehavior.py", line 14, in <module>
    from ndx_zebrafish import ZebrafishBehavior
  File "/home/vilimstih/phd/ndx-zebrafish/src/pynwb/ndx_zebrafish/__init__.py", line 22, in <module>
    load_namespaces(ndx_zebrafish_specpath)
  File "/home/vilimstih/miniconda3/envs/stytranwb/lib/python3.9/site-packages/hdmf/utils.py", line 587, in func_call
    return func(**pargs)
  File "/home/vilimstih/miniconda3/envs/stytranwb/lib/python3.9/site-packages/pynwb/__init__.py", line 102, in load_namespaces
    return __TYPE_MAP.load_namespaces(namespace_path)
  File "/home/vilimstih/miniconda3/envs/stytranwb/lib/python3.9/site-packages/hdmf/utils.py", line 583, in func_call
    return func(args[0], **pargs)
  File "/home/vilimstih/miniconda3/envs/stytranwb/lib/python3.9/site-packages/hdmf/build/manager.py", line 474, in load_namespaces
    deps = call_docval_func(self.__ns_catalog.load_namespaces, kwargs)
  File "/home/vilimstih/miniconda3/envs/stytranwb/lib/python3.9/site-packages/hdmf/utils.py", line 424, in call_docval_func
    return func(*fargs, **fkwargs)
  File "/home/vilimstih/miniconda3/envs/stytranwb/lib/python3.9/site-packages/hdmf/utils.py", line 583, in func_call
    return func(args[0], **pargs)
  File "/home/vilimstih/miniconda3/envs/stytranwb/lib/python3.9/site-packages/hdmf/spec/namespace.py", line 538, in load_namespaces
    ret[ns['name']] = self.__load_namespace(ns, reader, resolve=resolve)
  File "/home/vilimstih/miniconda3/envs/stytranwb/lib/python3.9/site-packages/hdmf/spec/namespace.py", line 460, in __load_namespace
    self.__register_type(ndt, inc_ns, catalog, registered_types)
  File "/home/vilimstih/miniconda3/envs/stytranwb/lib/python3.9/site-packages/hdmf/spec/namespace.py", line 470, in __register_type
    spec = inc_ns.get_spec(ndt)
  File "/home/vilimstih/miniconda3/envs/stytranwb/lib/python3.9/site-packages/hdmf/utils.py", line 583, in func_call
    return func(args[0], **pargs)
  File "/home/vilimstih/miniconda3/envs/stytranwb/lib/python3.9/site-packages/hdmf/spec/namespace.py", line 150, in get_spec
    raise ValueError("No specification for '%s' in namespace '%s'" % (data_type, self.name))
ValueError: No specification for 'ZebrafishBehavior' in namespace 'core'

Operating System

Linux

Python Executable

Python

Python Version

3.8

Package Versions

No response

Code of Conduct

[X] I agree to follow this project's Code of Conduct
[X] Have you checked the Contributing document?
[X] Have you ensured this bug was not already reported?

Hi @vilim ,

In your ndx-zebrafish.namespace.yaml file, you have written that ZebrafishBehavior is a neurodata type to include from the NWB core namespace. However, this is a type that you define in your extension; it does not exist in the core namespace.

To fix this, remove this line: https://github.com/nwb-zebrafish/ndx-zebrafish/blob/2f8b373b6cc93822ec71de651df11dc00c0bd530/src/spec/create_extension_spec.py#L29

When I removed this line and tried to import your extension, I found another error:

TypeError: NWBDatasetSpec.__init__: incorrect type for 'neurodata_type_inc' (got 'NWBGroupSpec', expected 'NWBDatasetSpec or str')

This is because your extension specifies datasets with the neurodata type "Orientation", "TailShape", and "Coordinates" which extend the neurodata type "SpatialSeries". However, "SpatialSeries" is a Group, not a Dataset, so these objects should be added to the "ZebrafishBehavior" neurodata type with the method add_group, not add_dataset.

Finally, we recommend as best practice, not defining a type in place where it is used - meaning do not use "neurodata_type_def" within add_group or add_dataset (I will add a warning to PyNWB to mention this anti-pattern). Instead, we recommend either:

define the neurodata types "Orientation", "TailShape", and "Coordinates" first, using NWBGroupSpec, just like you did for "ZebrafishBehavior" and remove "neurodata_type_def" from add_dataset (which should be changed to add_group per the above)
do not define new neurodata types "Orientation", "TailShape", and "Coordinates", since currently these are not any different than "SpatialSeries" except for the docstring. You can just include an object of type "SpatialSeries" with your docstring and without creating a new type. i.e. just remove the "neurodata_type_def" lines.

I would recommend option 2 here unless you plan to add additional attributes, datasets, groups, or links within "Orientation", "TailShape", and "Coordinates".

Please let me know if you have any questions.

NeurodataWithoutBorders / pynwb