Cannot print/repr an NWBFile -- ValueError: Not a dataset (not a dataset). (keywords?)

yarikoptic commented 4 years ago

Description

helper script:

$> cat ./print-pynwb.py 
#!/usr/bin/env python

import pynwb
import sys
print("pynwb: %s pynwb.__version__: %s" % (pynwb, pynwb.__version__))

for filename in sys.argv[1:]:
    with pynwb.NWBHDF5IO(filename, 'r', load_namespaces=True) as reader:
        nwbfile = reader.read()
    print(nwbfile)

Steps to Reproduce

wget/curl (or you could use datalad to install and get that or any other file within https://github.com/dandi/najafi-2018-nwb) the file
run the above script on it

$> python3 ./print-pynwb.py MD5E-s302896324--2a6346977e9584c3820fb1dc3e07c353.nwb 
pynwb: <module 'pynwb' from '/usr/lib/python3/dist-packages/pynwb/__init__.py'> pynwb.__version__: 1.0.3
/usr/lib/python3/dist-packages/hdmf/spec/namespace.py:452: UserWarning: ignoring namespace 'core' because it already exists
  warn("ignoring namespace '%s' because it already exists" % ns['name'])
Traceback (most recent call last):
  File "./print-pynwb.py", line 10, in <module>
    print(nwbfile)  
  File "/usr/lib/python3/dist-packages/pynwb/core.py", line 168, in __repr__
    if not hasattr(v, '__len__') or len(v) > 0:
  File "/usr/lib/python3/dist-packages/pynwb/core.py", line 1121, in __len__
    return len(self.id)
  File "/usr/lib/python3/dist-packages/pynwb/core.py", line 343, in __len__
    return len(self.__data)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "/usr/lib/python3/dist-packages/h5py/_hl/dataset.py", line 447, in __len__
    size = self.len()
  File "/usr/lib/python3/dist-packages/h5py/_hl/dataset.py", line 459, in len
    shape = self.shape
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "/usr/lib/python3/dist-packages/h5py/_hl/dataset.py", line 287, in shape
    return self.id.shape
  File "h5py/h5d.pyx", line 131, in h5py.h5d.DatasetID.shape.__get__
  File "h5py/h5d.pyx", line 132, in h5py.h5d.DatasetID.shape.__get__
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5d.pyx", line 288, in h5py.h5d.DatasetID.get_space
ValueError: Not a dataset (not a dataset)

I believe that file was generated using pynwb. All scripts are within aforementioned repo. Singularity image with the used pynwb etc is available via git-annex/datalad as well there.

Upon playing with minimalistic code, it was possible to reproduce it with a minimalistic example:

$> python3 another.py
/usr/lib/python3/dist-packages/hdmf/backends/hdf5/h5tools.py:96: UserWarning: No cached namespaces found in /tmp/testfile.nwb
  warnings.warn(msg)
Traceback (most recent call last):
  File "another.py", line 24, in <module>
    print(nwbfile)
  File "/usr/lib/python3/dist-packages/pynwb/core.py", line 168, in __repr__
    if not hasattr(v, '__len__') or len(v) > 0:
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "/usr/lib/python3/dist-packages/h5py/_hl/dataset.py", line 447, in __len__
    size = self.len()
  File "/usr/lib/python3/dist-packages/h5py/_hl/dataset.py", line 459, in len
    shape = self.shape
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "/usr/lib/python3/dist-packages/h5py/_hl/dataset.py", line 287, in shape
    return self.id.shape
  File "h5py/h5d.pyx", line 131, in h5py.h5d.DatasetID.shape.__get__
  File "h5py/h5d.pyx", line 132, in h5py.h5d.DatasetID.shape.__get__
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5d.pyx", line 288, in h5py.h5d.DatasetID.get_space
ValueError: Not a dataset (not a dataset)

$> cat another.py
import os
from datetime import datetime
from dateutil.tz import tzlocal, tzutc

from pynwb import NWBFile, TimeSeries
from pynwb import NWBHDF5IO
from pynwb.file import Subject, ElectrodeTable
from pynwb.epoch import TimeIntervals

filename ="/tmp/testfile.nwb"
nwbfile = NWBFile('session description',
                  'identifier',
                  datetime.now(tzlocal()),
                  file_create_date=datetime.now(tzlocal()),
                  lab='a Lab',
                  # Keywords cause that puke upon `repr` ValueError: Not a dataset (not a dataset)
                  keywords=('these', 'are', 'keywords')
                 )
with NWBHDF5IO(filename, 'w') as io:
    io.write(nwbfile, cache_spec=False)

with NWBHDF5IO(filename, 'r', load_namespaces=True) as reader:
    nwbfile = reader.read()
print(nwbfile)

Having keywords there is a trigger (comment out and see it pass)

Environment

Python Executable: Python
Python Version: 3.7
Operating System: Linux
HDMF Version: 1.1.2
Version of PyNWB used: 1.0.3

Checklist

[x] Have you ensured the feature or change was not already reported?
[x] Have you included a brief and descriptive title?
[x] Have you included a clear description of the problem you are trying to solve?
[x] Have you included a minimal code snippet that reproduces the issue you are encountering?
[ ] Have you checked our Contributing document?

rly commented 4 years ago

@yarikoptic Thanks for the bug report. I can reproduce the bug with PyNWB 1.0.3 and HDMF 1.1.2. Could you try this with PyNWB 1.1.0 installed or the dev branch of PyNWB installed? Your MWE runs fine using the dev branch of PyNWB on my machine.

yarikoptic commented 4 years ago

eh, apparently the file url was butchered. I have adjusted it -- should be downloadable now.

With 1.1.1 -- the same.

with 1.1.0 -- even "worse"

```shell (dev3) yoh@ip-172-31-33-190:/tmp$ python ~/print-pynwb.py MD5E-s302896324--2a6346977e9584c3820fb1dc3e07c353.nwb pynwb: pynwb.__version__: 1.1.0 /home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/hdmf/spec/namespace.py:457: UserWarning: ignoring namespace 'core' because it already exists warn("ignoring namespace '%s' because it already exists" % ns['name']) Traceback (most recent call last): File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/hdmf/build/map.py", line 1207, in construct obj.__init__(**kwargs) File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/hdmf/utils.py", line 438, in func_call return func(self, **parsed['args']) File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/pynwb/core.py", line 351, in __init__ call_docval_func(super(VectorData, self).__init__, kwargs) File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/hdmf/utils.py", line 327, in call_docval_func return func(*fargs, **fkwargs) File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/hdmf/utils.py", line 438, in func_call return func(self, **parsed['args']) File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/pynwb/core.py", line 281, in __init__ call_docval_func(super(NWBData, self).__init__, kwargs) File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/hdmf/utils.py", line 327, in call_docval_func return func(*fargs, **fkwargs) File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/hdmf/utils.py", line 438, in func_call return func(self, **parsed['args']) File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/pynwb/core.py", line 34, in __init__ call_docval_func(super(NWBBaseType, self).__init__, kwargs) File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/hdmf/utils.py", line 327, in call_docval_func return func(*fargs, **fkwargs) File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/hdmf/utils.py", line 436, in func_call raise_from(ExceptionType(msg), None) File "", line 3, in raise_from TypeError: None is not allowed for 'data' (expected 'ndarray, list, tuple, Dataset, HDMFDataset or data', not None) The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/home/yoh/print-pynwb.py", line 9, in nwbfile = reader.read() File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/hdmf/backends/hdf5/h5tools.py", line 293, in read return call_docval_func(super(HDF5IO, self).read, kwargs) File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/hdmf/utils.py", line 327, in call_docval_func return func(*fargs, **fkwargs) File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/hdmf/utils.py", line 438, in func_call return func(self, **parsed['args']) File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/hdmf/backends/io.py", line 35, in read container = self.__manager.construct(f_builder) File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/hdmf/utils.py", line 438, in func_call return func(self, **parsed['args']) File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/hdmf/build/map.py", line 208, in construct result = self.__type_map.construct(builder, self, None) File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/hdmf/utils.py", line 438, in func_call return func(self, **parsed['args']) File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/hdmf/build/map.py", line 1712, in construct return attr_map.construct(builder, build_manager, parent) File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/hdmf/utils.py", line 438, in func_call return func(self, **parsed['args']) File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/hdmf/build/map.py", line 1175, in construct subspecs = self.__get_subspec_values(builder, self.spec, manager) File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/hdmf/build/map.py", line 1117, in __get_subspec_values self.__get_sub_builders(groups, spec.groups, manager, ret) File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/hdmf/build/map.py", line 1155, in __get_sub_builders ret.update(self.__get_subspec_values(sub_builder, subspec, manager)) File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/hdmf/build/map.py", line 1117, in __get_subspec_values self.__get_sub_builders(groups, spec.groups, manager, ret) File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/hdmf/build/map.py", line 1147, in __get_sub_builders sub_builder = self.__flatten(sub_builder, subspec, manager) File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/hdmf/build/map.py", line 1160, in __flatten tmp = [manager.construct(b) for b in sub_builder] File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/hdmf/build/map.py", line 1160, in tmp = [manager.construct(b) for b in sub_builder] File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/hdmf/utils.py", line 438, in func_call return func(self, **parsed['args']) File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/hdmf/build/map.py", line 204, in construct result = self.__type_map.construct(builder, self, parent) File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/hdmf/utils.py", line 438, in func_call return func(self, **parsed['args']) File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/hdmf/build/map.py", line 1712, in construct return attr_map.construct(builder, build_manager, parent) File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/hdmf/utils.py", line 438, in func_call return func(self, **parsed['args']) File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/hdmf/build/map.py", line 1175, in construct subspecs = self.__get_subspec_values(builder, self.spec, manager) File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/hdmf/build/map.py", line 1117, in __get_subspec_values self.__get_sub_builders(groups, spec.groups, manager, ret) File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/hdmf/build/map.py", line 1147, in __get_sub_builders sub_builder = self.__flatten(sub_builder, subspec, manager) File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/hdmf/build/map.py", line 1160, in __flatten tmp = [manager.construct(b) for b in sub_builder] File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/hdmf/build/map.py", line 1160, in tmp = [manager.construct(b) for b in sub_builder] File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/hdmf/utils.py", line 438, in func_call return func(self, **parsed['args']) File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/hdmf/build/map.py", line 204, in construct result = self.__type_map.construct(builder, self, parent) File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/hdmf/utils.py", line 438, in func_call return func(self, **parsed['args']) File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/hdmf/build/map.py", line 1712, in construct return attr_map.construct(builder, build_manager, parent) File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/hdmf/utils.py", line 438, in func_call return func(self, **parsed['args']) File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/hdmf/build/map.py", line 1175, in construct subspecs = self.__get_subspec_values(builder, self.spec, manager) File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/hdmf/build/map.py", line 1117, in __get_subspec_values self.__get_sub_builders(groups, spec.groups, manager, ret) File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/hdmf/build/map.py", line 1147, in __get_sub_builders sub_builder = self.__flatten(sub_builder, subspec, manager) File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/hdmf/build/map.py", line 1160, in __flatten tmp = [manager.construct(b) for b in sub_builder] File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/hdmf/build/map.py", line 1160, in tmp = [manager.construct(b) for b in sub_builder] File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/hdmf/utils.py", line 438, in func_call return func(self, **parsed['args']) File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/hdmf/build/map.py", line 204, in construct result = self.__type_map.construct(builder, self, parent) File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/hdmf/utils.py", line 438, in func_call return func(self, **parsed['args']) File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/hdmf/build/map.py", line 1712, in construct return attr_map.construct(builder, build_manager, parent) File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/hdmf/utils.py", line 438, in func_call return func(self, **parsed['args']) File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/hdmf/build/map.py", line 1175, in construct subspecs = self.__get_subspec_values(builder, self.spec, manager) File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/hdmf/build/map.py", line 1118, in __get_subspec_values self.__get_sub_builders(datasets, spec.datasets, manager, ret) File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/hdmf/build/map.py", line 1157, in __get_sub_builders ret[subspec] = manager.construct(sub_builder) File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/hdmf/utils.py", line 438, in func_call return func(self, **parsed['args']) File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/hdmf/build/map.py", line 204, in construct result = self.__type_map.construct(builder, self, parent) File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/hdmf/utils.py", line 438, in func_call return func(self, **parsed['args']) File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/hdmf/build/map.py", line 1712, in construct return attr_map.construct(builder, build_manager, parent) File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/hdmf/utils.py", line 438, in func_call return func(self, **parsed['args']) File "/home/yoh/dandi-cli/venvs/dev3/lib/python3.6/site-packages/hdmf/build/map.py", line 1210, in construct raise_from(Exception(msg), ex) File "", line 3, in raise_from Exception: Could not construct VectorData object ```

rly commented 4 years ago

@yarikoptic In your MWE, please move your print(nwbfile) statement to be within the with file context manager. The file needs to be open because the contents are read lazily.

Doing that made your code work for me with PyNWB 1.1.1.

yarikoptic commented 4 years ago

@rly THANKS! confirming -- works for me too! Having said that, repr shouldn't fail either within or outside of the context handler. Since repr here is not really a "string representation of an object" here anyways, I would recommend to fix this issue but making repr not crash even if file is no longer open (outside of the context manager) by adding N/A - closed file to those fields value for which is no longer available. It would make it (more) obvious of naive new users like me what I have done "wrong".

But getting back to keywords specifically - since it is not that heavy of an object, it might be worth making it loaded non-lazily.

NeurodataWithoutBorders / pynwb