search

NeurodataWithoutBorders / pynwb

A Python API for working with Neurodata stored in the NWB Format
https://pynwb.readthedocs.io
Other
176 stars 85 forks source link

Provide dedicated "Cannot load NWB 1.0" Exception/Error instead of an arbitrary crash #1077

Open yarikoptic opened 5 years ago

yarikoptic commented 5 years ago

I think I have ran into the now closed https://github.com/NeurodataWithoutBorders/pynwb/issues/1051 while trying to read an example file from exp2nwb: https://github.com/NeurodataWithoutBorders/exp2nwb/blob/master/example/Example.nwb?raw=true

$> wget 'https://github.com/NeurodataWithoutBorders/exp2nwb/blob/master/example/Example.nwb?raw=true'
--2019-09-28 12:46:29--  https://github.com/NeurodataWithoutBorders/exp2nwb/blob/master/example/Example.nwb?raw=true
Resolving github.com (github.com)... 140.82.113.3
Connecting to github.com (github.com)|140.82.113.3|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://github.com/NeurodataWithoutBorders/exp2nwb/raw/master/example/Example.nwb [following]
--2019-09-28 12:46:29--  https://github.com/NeurodataWithoutBorders/exp2nwb/raw/master/example/Example.nwb
Reusing existing connection to github.com:443.
HTTP request sent, awaiting response... 302 Found
Location: https://raw.githubusercontent.com/NeurodataWithoutBorders/exp2nwb/master/example/Example.nwb [following]
--2019-09-28 12:46:29--  https://raw.githubusercontent.com/NeurodataWithoutBorders/exp2nwb/master/example/Example.nwb
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.200.133
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.200.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 180819 (177K) [application/octet-stream]
Saving to: ‘Example.nwb?raw=true’

Example.nwb?raw=true                                     100%[================================================================================================================================>] 176.58K  --.-KB/s    in 0.1s    

2019-09-28 12:46:29 (1.32 MB/s) - ‘Example.nwb?raw=true’ saved [180819/180819]

$> python3 ./print-pynwb.py Example.nwb\?raw=true 
pynwb: <module 'pynwb' from '/usr/lib/python3/dist-packages/pynwb/__init__.py'> pynwb.__version__: 1.0.3
/usr/lib/python3/dist-packages/hdmf/backends/hdf5/h5tools.py:96: UserWarning: No cached namespaces found in Example.nwb?raw=true
  warnings.warn(msg)
Traceback (most recent call last):
  File "./print-pynwb.py", line 9, in <module>
    nwbfile = reader.read()
  File "/usr/lib/python3/dist-packages/hdmf/backends/hdf5/h5tools.py", line 246, in read
    return call_docval_func(super(HDF5IO, self).read, kwargs)
  File "/usr/lib/python3/dist-packages/hdmf/utils.py", line 325, in call_docval_func
    return func(*fargs, **fkwargs)
  File "/usr/lib/python3/dist-packages/hdmf/utils.py", line 436, in func_call
    return func(self, **parsed['args'])
  File "/usr/lib/python3/dist-packages/hdmf/backends/io.py", line 35, in read
    container = self.__manager.construct(f_builder)
  File "/usr/lib/python3/dist-packages/hdmf/utils.py", line 436, in func_call
    return func(self, **parsed['args'])
  File "/usr/lib/python3/dist-packages/hdmf/build/map.py", line 207, in construct
    result = self.__type_map.construct(builder, self, None)
  File "/usr/lib/python3/dist-packages/hdmf/utils.py", line 436, in func_call
    return func(self, **parsed['args'])
  File "/usr/lib/python3/dist-packages/hdmf/build/map.py", line 1668, in construct
    attr_map = self.get_map(builder)
  File "/usr/lib/python3/dist-packages/hdmf/utils.py", line 436, in func_call
    return func(self, **parsed['args'])
  File "/usr/lib/python3/dist-packages/hdmf/build/map.py", line 1595, in get_map
    container_cls = self.get_cls(obj)
  File "/usr/lib/python3/dist-packages/hdmf/utils.py", line 436, in func_call
    return func(self, **parsed['args'])
  File "/usr/lib/python3/dist-packages/hdmf/build/map.py", line 1522, in get_cls
    raise ValueError("No data_type found for builder %s" % builder.path)
ValueError: No data_type found for builder root

If that is the best way to determine that the file is NWB 1.0 not 2.0, please consider catching it around h5tools.py", line 246, in read and reraising a dedicated/descriptive exception. If there is a better way (I hope) then please add it that sensing even before even trying to load an incompatible .nwb file.

yarikoptic commented 5 years ago
One more example where I was told that file is probably NWB 1.0 ```shell (git)smaug:/mnt/btrfs/datasets/datalad/crawl/crcns/ssc-7[master]git $> dandi ls data/L4E_whole_cell/Exp_2015-09-05_001_0001-0162.nwb Traceback (most recent call last): File "/home/yoh/proj/dandi/dandi-cli/venvs/dev3/bin/dandi", line 11, in load_entry_point('dandi', 'console_scripts', 'dandi')() File "/usr/lib/python3/dist-packages/click/core.py", line 764, in __call__ return self.main(*args, **kwargs) File "/usr/lib/python3/dist-packages/click/core.py", line 717, in main rv = self.invoke(ctx) File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "/usr/lib/python3/dist-packages/click/core.py", line 956, in invoke return ctx.invoke(self.callback, **ctx.params) File "/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke return callback(*args, **kwargs) File "/home/yoh/proj/dandi/dandi-cli/dandi/cli/command.py", line 169, in ls rec.update(get_metadata_pyout(path)) File "/home/yoh/proj/dandi/dandi-cli/dandi/cli/command.py", line 61, in get_metadata_pyout meta = get_metadata(path) File "/home/yoh/proj/dandi/dandi-cli/dandi/pynwb_utils.py", line 30, in get_metadata nwb = io.read() File "/home/yoh/proj/dandi/dandi-cli/venvs/dev3/lib/python3.7/site-packages/hdmf/backends/hdf5/h5tools.py", line 293, in read return call_docval_func(super(HDF5IO, self).read, kwargs) File "/home/yoh/proj/dandi/dandi-cli/venvs/dev3/lib/python3.7/site-packages/hdmf/utils.py", line 327, in call_docval_func return func(*fargs, **fkwargs) File "/home/yoh/proj/dandi/dandi-cli/venvs/dev3/lib/python3.7/site-packages/hdmf/utils.py", line 438, in func_call return func(self, **parsed['args']) File "/home/yoh/proj/dandi/dandi-cli/venvs/dev3/lib/python3.7/site-packages/hdmf/backends/io.py", line 31, in read f_builder = self.read_builder() File "/home/yoh/proj/dandi/dandi-cli/venvs/dev3/lib/python3.7/site-packages/hdmf/utils.py", line 438, in func_call return func(self, **parsed['args']) File "/home/yoh/proj/dandi/dandi-cli/venvs/dev3/lib/python3.7/site-packages/hdmf/backends/hdf5/h5tools.py", line 308, in read_builder f_builder = self.__read_group(self.__file, ROOT_NAME, ignore=ignore) File "/home/yoh/proj/dandi/dandi-cli/venvs/dev3/lib/python3.7/site-packages/hdmf/backends/hdf5/h5tools.py", line 385, in __read_group builder = read_method(sub_h5obj) File "/home/yoh/proj/dandi/dandi-cli/venvs/dev3/lib/python3.7/site-packages/hdmf/backends/hdf5/h5tools.py", line 385, in __read_group builder = read_method(sub_h5obj) File "/home/yoh/proj/dandi/dandi-cli/venvs/dev3/lib/python3.7/site-packages/hdmf/backends/hdf5/h5tools.py", line 385, in __read_group builder = read_method(sub_h5obj) File "/home/yoh/proj/dandi/dandi-cli/venvs/dev3/lib/python3.7/site-packages/hdmf/backends/hdf5/h5tools.py", line 393, in __read_group ret = GroupBuilder(name, **kwargs) File "/home/yoh/proj/dandi/dandi-cli/venvs/dev3/lib/python3.7/site-packages/hdmf/utils.py", line 438, in func_call return func(self, **parsed['args']) File "/home/yoh/proj/dandi/dandi-cli/venvs/dev3/lib/python3.7/site-packages/hdmf/build/builders.py", line 179, in __init__ self.set_dataset(dataset) File "/home/yoh/proj/dandi/dandi-cli/venvs/dev3/lib/python3.7/site-packages/hdmf/utils.py", line 438, in func_call return func(self, **parsed['args']) File "/home/yoh/proj/dandi/dandi-cli/venvs/dev3/lib/python3.7/site-packages/hdmf/build/builders.py", line 289, in set_dataset self.__set_builder(builder, GroupBuilder.__dataset) File "/home/yoh/proj/dandi/dandi-cli/venvs/dev3/lib/python3.7/site-packages/hdmf/build/builders.py", line 259, in __set_builder (name, self.obj_type[name], self.name, obj_type)) KeyError: "'electrode_name' already exists as attributes in membrane_potential, cannot set as datasets" ```

but the exception (KeyError: "'electrode_name' already exists as attributes in membrane_potential...) had no hint on that

yarikoptic commented 5 years ago

aha -- nwb-schema talks about /specifications/<namespace-name>/<version>/...,

and my sample NWB 2.0 file does carry a core/2.0.1 group ```shell $> h5ls -S -r mouse1_fni16_150818_001_ch2-PnevPanResults-170808-180842.nwb/specifications/ /core Group /core/2.0.2 Group /core/2.0.2/namespace Dataset {SCALAR} /core/2.0.2/nwb.base Dataset {SCALAR} /core/2.0.2/nwb.behavior Dataset {SCALAR} /core/2.0.2/nwb.ecephys Dataset {SCALAR} /core/2.0.2/nwb.epoch Dataset {SCALAR} /core/2.0.2/nwb.file Dataset {SCALAR} /core/2.0.2/nwb.icephys Dataset {SCALAR} /core/2.0.2/nwb.image Dataset {SCALAR} /core/2.0.2/nwb.misc Dataset {SCALAR} /core/2.0.2/nwb.ogen Dataset {SCALAR} /core/2.0.2/nwb.ophys Dataset {SCALAR} /core/2.0.2/nwb.retinotopy Dataset {SCALAR} ```
while a sample file in ssc-7 dataset has none: ```shell $> h5ls -S -r data/L4E_whole_cell/Exp_2015-09-05_001_0001-0162.nwb/specifications/ specifications/**NOT FOUND** % ```

from glancing over that docs/storage/source/storage_hdf5.rst it is not 100% clear to me if that is a feature of NWB:N specifically or NWB in general, and when was it introduced?

Also the https://github.com/NeurodataWithoutBorders/pynwb/blob/dev/README.rst starts with "It provides a high-level API for efficiently working with Neurodata stored in the NWB format." - it seems not to limit to version of NWB 2.0 or above. If NWB prior 2.0 are not supposed to be supported, it should be clarified in the README.rst IMHO.

Unfortunately I am still learning NWB ecosystem so cannot really propose any definitive solution/PR here without guidance ;)

bendichter commented 5 years ago

@yarikoptic can we take this over to slack? I'm happy to help you sort these issues out and open this file in pynwb. It looks like this file is in 2.0 after all, so not relevant to this issue.

yarikoptic commented 5 years ago

@yarikoptic can we take this over to slack? I'm happy to help you sort these issues out and open this file in pynwb. It looks like this file is in 2.0 after all, so not relevant to this issue.

and we did (in jitsi), summary: me