Are namespaces/extensions cached in example files?

[pgleeson]

Testing with latest generated examples on OneDrive. Should I be able to just load using:

io = NWBHDF5IO(nwb_file, 'r', load_namespaces=True)

I'm getting the following error:

/home/padraig/anaconda2/envs/silvernwb/lib/python3.7/site-packages/hdmf-1.6.0-py3.7.egg/hdmf/spec/namespace.py:456: UserWarning: ignoring namespace 'hdmf-common' because it already exists
  warn("ignoring namespace '%s' because it already exists" % ns['name'])
/home/padraig/anaconda2/envs/silvernwb/lib/python3.7/site-packages/hdmf-1.6.0-py3.7.egg/hdmf/spec/namespace.py:456: UserWarning: ignoring namespace 'core' because it already exists
  warn("ignoring namespace '%s' because it already exists" % ns['name'])
Error loading via pynwb!
Cannot dynamically generate class for type 'Unknown'. Type 'ZplanePockelsDataset' does not exist. Please define that type before defining 'Unknown'.

[pgleeson]

@ageorgou Any thoughts on this?

[ageorgou]

@pgleeson Looking into this now. The namespaces are cached in the file, but I think there is a bug in one of the custom types. Will update when fixed.

[alessandrofelder]

@pgleeson we have found a work-around at least:

io = NWBHDF5IO("C:\\Users\\Alessandro\\Documents\\UCL-projects\\silverlab\\Data\\nwb2\\sample_pointing_videos_161215_15_34_21.nwb" ,
'r', extensions = 'src\\silverlabnwb\\silverlab.namespace.yaml')
io.read()

this requires the path to an additional file "silverlab.namespace.yaml" found in the GitHub repo under src\\silverlabnwb\\, which defines the custom silverlab extension. Remember to adapt the direction of the slashes "\","/" to your OS. Hope this helps!

[ageorgou]

@pgleeson Did the above workaround work for you? We've found this to also work (adjusting for where you're running it from):

from pynwb import get_class, load_namespaces, NWBHDF5IO

load_namespaces('src/silverlabnwb/silverlab.namespace.yaml')
get_class('ZplanePockelsDataset', 'silverlab_extended_schema')
io = NWBHDF5IO(nwb_file, 'r')

Essentially we need to load the definition of that one class first, or the namespace won't be reconstructed correctly. We're still trying to find out whether this is because we are doing something wrong or an issue with the extension mechanism.

[pgleeson]

@ageorgou Thanks, that new suggestion worked. I've managed to load one of your example NWB2 file locally and view it with a slightly modified NWBExplorer (see below). Will do some more tests on other files over the next few days, and then arrange a meeting together with @jonc125 for a general update.

[ageorgou]

Confirmed to be an issue with the pynwb backend. Will be fixed in hdmf-dev/hdmf#322.

[ageorgou]

Also see an alternative way of structuring the extension which should work with the current version of pynwb/hdmf.

[ageorgou]

Update: the resolution has been in the development version of hdmf since March 26, but there hasn't been a new release yet, so it won't be automatically included even when installing the latest pynwb (version 1.3).

[ageorgou]

The latest version requires pynwb 1.3.2, which should read the stored namespaces directly without the workaround above. We'll make a release soon, but for now it's available in master (currently 3c5816c87e23569002187b9ce096113487ef797b).

@pgleeson We'll wait for you to confirm that you can read the files before closing.

[pgleeson]

Great! Will try to try this out before our meeting next week on Friday

[pgleeson]

As mentioned on Friday, works fine for me with namespace cached. Thnaks.