demo code causes error: unresolved reference on 0.2.0 and on current dev branch

bendichter commented 6 years ago

I'm new to the project so sorry if this is a known issue. I'm trying to get up and running but the demo code is giving me an error. I get the same error working with my own data.

Steps to Reproduce

The following is taken from https://pynwb.readthedocs.io/en/latest/example.html:

import numpy as np

rate = 10.0
np.random.seed(1234)
data_len = 1000
ephys_data = np.random.rand(data_len)
ephys_timestamps = np.arange(data_len) / rate
spatial_timestamps = ephys_timestamps[::10]
spatial_data = np.cumsum(np.random.normal(size=(2, len(spatial_timestamps))), axis=-1).T

from datetime import datetime
from pynwb import NWBFile

f = NWBFile('the PyNWB tutorial', 'my first synthetic recording', 'EXAMPLE_ID', datetime.now(),
            experimenter='Dr. Bilbo Baggins',
            lab='Bag End Laboratory',
            institution='University of Middle Earth at the Shire',
            experiment_description='I went on an adventure with thirteen dwarves to reclaim vast treasures.',
            session_id='LONELYMTN')

epoch_tags = ('example_epoch',)

ep1 = f.create_epoch(source='an hypothetical source', name='epoch1', start=0.0, stop=1.0,
                     tags=epoch_tags,
                     description="the first test epoch")

ep2 = f.create_epoch(source='an hypothetical source', name='epoch2', start=0.0, stop=1.0,
                     tags=epoch_tags,
                     description="the second test epoch")

device = f.create_device(name='trodes_rig123', source="a source")

electrode_name = 'tetrode1'
source = "an hypothetical source"
description = "an example tetrode"
location = "somewhere in the hippocampus"

electrode_group = f.create_electrode_group(electrode_name,
                                           source=source,
                                           description=description,
                                           location=location,
                                           device=device)

from pynwb.ecephys import ElectrodeTable, ElectrodeTableRegion

electrode_table = ElectrodeTable('electrodes')
for idx in [1, 2, 3, 4]:
    electrode_table.add_row(idx,
                            x=1.0, y=2.0, z=3.0,
                            imp=float(-idx),
                            location='CA1', filtering='none',
                            description='channel %s' % idx, group=electrode_group)

electrode_table_region = ElectrodeTableRegion(electrode_table, [0, 2], 'the first and third electrodes')

from pynwb.ecephys import ElectricalSeries
from pynwb.behavior import SpatialSeries

ephys_ts = ElectricalSeries('test_ephys_data',
                            'an hypothetical source',
                            ephys_data,
                            electrode_table_region,
                            timestamps=ephys_timestamps,
                            # Alternatively, could specify starting_time and rate as follows
                            # starting_time=ephys_timestamps[0],
                            # rate=rate,
                            resolution=0.001,
                            comments="This data was randomly generated with numpy, using 1234 as the seed",
                            description="Random numbers generated with numpy.random.rand")
f.add_acquisition(ephys_ts, [ep1, ep2])

spatial_ts = SpatialSeries('test_spatial_timeseries',
                           'a stumbling rat',
                           spatial_data,
                           'origin on x,y-plane',
                           timestamps=spatial_timestamps,
                           resolution=0.1,
                           comments="This data was generated with numpy, using 1234 as the seed",
                           description="This 2D Brownian process generated with "
                                       "np.cumsum(np.random.normal(size=(2, len(spatial_timestamps))), axis=-1).T")
f.add_acquisition(spatial_ts, [ep1, ep2])

from pynwb import get_manager
from pynwb.form.backends.hdf5 import HDF5IO

filename = "example.h5"
io = HDF5IO(filename, manager=get_manager(), mode='w')
io.write(f)
io.close()

It produces the error:

Traceback (most recent call last):
  File "/Users/bendichter/Development/pynwb/src/pynwb/form/backends/hdf5/h5tools.py", line 286, in __add_refs
    call()
  File "/Users/bendichter/Development/pynwb/src/pynwb/form/backends/hdf5/h5tools.py", line 499, in _filler
    ref = self.__get_ref(data, builder.region)
  File "/Users/bendichter/Development/pynwb/src/pynwb/form/backends/hdf5/h5tools.py", line 623, in __get_ref
    dset = self.__file[path]
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "/anaconda/envs/preprocessing/lib/python3.6/site-packages/h5py/_hl/group.py", line 167, in __getitem__
    oid = h5o.open(self.id, self._e(name), lapl=self._lapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5o.pyx", line 190, in h5py.h5o.open
KeyError: "Unable to open object (object 'electrodes' doesn't exist)"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/bendichter/Development/ecog_preprocessing/nwb_demo.py", line 90, in <module>
    io.write(f)
  File "/Users/bendichter/Development/pynwb/src/pynwb/form/utils.py", line 306, in func_call
    return func(self, **parsed['args'])
  File "/Users/bendichter/Development/pynwb/src/pynwb/form/backends/hdf5/h5tools.py", line 123, in write
    call_docval_func(super(HDF5IO, self).write, kwargs)
  File "/Users/bendichter/Development/pynwb/src/pynwb/form/utils.py", line 212, in call_docval_func
    return func(*fargs, **fkwargs)
  File "/Users/bendichter/Development/pynwb/src/pynwb/form/utils.py", line 306, in func_call
    return func(self, **parsed['args'])
  File "/Users/bendichter/Development/pynwb/src/pynwb/form/backends/io.py", line 39, in write
    self.write_builder(f_builder)
  File "/Users/bendichter/Development/pynwb/src/pynwb/form/utils.py", line 306, in func_call
    return func(self, **parsed['args'])
  File "/Users/bendichter/Development/pynwb/src/pynwb/form/backends/hdf5/h5tools.py", line 271, in write_builder
    self.__add_refs()
  File "/Users/bendichter/Development/pynwb/src/pynwb/form/backends/hdf5/h5tools.py", line 289, in __add_refs
    raise RuntimeError('Unable to resolve reference')
RuntimeError: Unable to resolve reference

in pynwb 0.2.0 the error message is a little more straightforward:

Traceback (most recent call last):
  File "/Users/bendichter/Development/ecog_preprocessing/nwb_demo.py", line 75, in <module>
    io.write(f)
  File "/anaconda/envs/preprocessing/lib/python3.6/site-packages/pynwb/form/utils.py", line 306, in func_call
    return func(self, **parsed['args'])
  File "/anaconda/envs/preprocessing/lib/python3.6/site-packages/pynwb/form/backends/io.py", line 34, in write
    self.write_builder(f_builder)
  File "/anaconda/envs/preprocessing/lib/python3.6/site-packages/pynwb/form/utils.py", line 306, in func_call
    return func(self, **parsed['args'])
  File "/anaconda/envs/preprocessing/lib/python3.6/site-packages/pynwb/form/backends/hdf5/h5tools.py", line 177, in write_builder
    self.__add_refs()
  File "/anaconda/envs/preprocessing/lib/python3.6/site-packages/pynwb/form/backends/hdf5/h5tools.py", line 190, in __add_refs
    call()
  File "/anaconda/envs/preprocessing/lib/python3.6/site-packages/pynwb/form/backends/hdf5/h5tools.py", line 364, in _filler
    ref = self.__get_ref(data, builder.region)
  File "/anaconda/envs/preprocessing/lib/python3.6/site-packages/pynwb/form/backends/hdf5/h5tools.py", line 499, in __get_ref
    dset = self.__file[path]
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "/anaconda/envs/preprocessing/lib/python3.6/site-packages/h5py/_hl/group.py", line 167, in __getitem__
    oid = h5o.open(self.id, self._e(name), lapl=self._lapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5o.pyx", line 190, in h5py.h5o.open
KeyError: "Unable to open object (object 'electrodes' doesn't exist)"

The code runs through if f.add_acquisition(ephys_ts, [ep1, ep2]) is commented out, so it appears to be an issue with adding an ElectricalSeries object. Interestingly, SpatialSeries does not appear to have this problem. It can be kept in without causing the unresolved reference error.

Environment

Please describe your environment according to the following bullet points.

Python Executable: Conda
Python Version: Python 3.6
Operating System: macOS
Pynwb Version: Version of Pynwb used: 0.2.0 and dev branch

nicain commented 6 years ago

@bendichter Hi Ben, thanks for submitting the report, it is really appreciated! Response on this may be a little slow as the team gets up-and-running again after the holiday break, but we will def. address this.

I have reproduced the exception on my environment (Ubuntu 16.04.2). @ajtritt I am assigning this to you, but I will poke at it and see if I can find something going on; feel free to assign someone else if you need to.

nicain commented 6 years ago

Looks like the builder parent is not getting set to acquisition. Still working on figuring out why...

bendichter commented 6 years ago

Thanks for looking into this! Would it be possible to add the demo code as a test to avoid this in the future?

dorukozturk commented 6 years ago

:+1: for that idea. I think it is a good idea to add tests for the bugs that we encounter.

bendichter commented 6 years ago

I'm still seeing this error when I run the demo code:

import numpy as np
from datetime import datetime
from pynwb import NWBFile
from pynwb.ecephys import ElectrodeTable, ElectrodeTableRegion
from pynwb.ecephys import ElectricalSeries
from pynwb import get_manager
from pynwb.form.backends.hdf5 import HDF5IO

rate = 10.0
np.random.seed(1234)
data_len = 1000
ephys_data = np.random.rand(data_len)
ephys_timestamps = np.arange(data_len) / rate
spatial_timestamps = ephys_timestamps[::10]
spatial_data = np.cumsum(np.random.normal(size=(2, len(spatial_timestamps))), axis=-1).T

f = NWBFile('the PyNWB tutorial', 'my first synthetic recording', 'EXAMPLE_ID', datetime.now(),
            experimenter='Dr. Bilbo Baggins',
            lab='Bag End Laboratory',
            institution='University of Middle Earth at the Shire',
            experiment_description='I went on an adventure with thirteen dwarves to reclaim vast treasures.',
            session_id='LONELYMTN')

epoch_tags = ('example_epoch',)

ep1 = f.create_epoch(source='an hypothetical source', name='epoch1', start=0.0, stop=1.0,
                     tags=epoch_tags,
                     description="the first test epoch")

device = f.create_device(name='trodes_rig123', source="a source")

electrode_name = 'tetrode1'
source = "an hypothetical source"
description = "an example tetrode"
location = "somewhere in the hippocampus"

electrode_group = f.create_electrode_group(electrode_name,
                                           source=source,
                                           description=description,
                                           location=location,
                                           device=device)

electrode_table = ElectrodeTable('electrodes')
for idx in [1, 2, 3, 4]:
    electrode_table.add_row(idx,
                            x=1.0, y=2.0, z=3.0,
                            imp=float(-idx),
                            location='CA1', filtering='none',
                            description='channel %s' % idx, group=electrode_group)

electrode_table_region = ElectrodeTableRegion(electrode_table, [0, 2], 'the first and third electrodes')

ephys_ts = ElectricalSeries('test_ephys_data',
                            'an hypothetical source',
                            ephys_data,
                            electrode_table_region,
                            timestamps=ephys_timestamps,
                            # Alternatively, could specify starting_time and rate as follows
                            # starting_time=ephys_timestamps[0],
                            # rate=rate,
                            resolution=0.001,
                            comments="This data was randomly generated with numpy, using 1234 as the seed",
                            description="Random numbers generated with numpy.random.rand")
f.add_acquisition(ephys_ts, [ep1])

filename = "example.h5"
io = HDF5IO(filename, manager=get_manager(), mode='w')
io.write(f)
io.close()

Traceback (most recent call last):
  File "/Users/bendichter/Development/pynwb/src/pynwb/form/backends/hdf5/h5tools.py", line 284, in __add_refs
    call()
  File "/Users/bendichter/Development/pynwb/src/pynwb/form/backends/hdf5/h5tools.py", line 494, in _filler
    ref = self.__get_ref(data, builder.region)
  File "/Users/bendichter/Development/pynwb/src/pynwb/form/backends/hdf5/h5tools.py", line 618, in __get_ref
    dset = self.__file[path]
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "/anaconda/envs/preprocessing/lib/python3.6/site-packages/h5py/_hl/group.py", line 167, in __getitem__
    oid = h5o.open(self.id, self._e(name), lapl=self._lapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5o.pyx", line 190, in h5py.h5o.open
KeyError: "Unable to open object (object 'electrodes' doesn't exist)"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/bendichter/Development/ecog_preprocessing/nwb_demo.py", line 71, in <module>
    io.write(f)
  File "/Users/bendichter/Development/pynwb/src/pynwb/form/utils.py", line 306, in func_call
    return func(self, **parsed['args'])
  File "/Users/bendichter/Development/pynwb/src/pynwb/form/backends/hdf5/h5tools.py", line 121, in write
    call_docval_func(super(HDF5IO, self).write, kwargs)
  File "/Users/bendichter/Development/pynwb/src/pynwb/form/utils.py", line 212, in call_docval_func
    return func(*fargs, **fkwargs)
  File "/Users/bendichter/Development/pynwb/src/pynwb/form/utils.py", line 306, in func_call
    return func(self, **parsed['args'])
  File "/Users/bendichter/Development/pynwb/src/pynwb/form/backends/io.py", line 39, in write
    self.write_builder(f_builder)
  File "/Users/bendichter/Development/pynwb/src/pynwb/form/utils.py", line 306, in func_call
    return func(self, **parsed['args'])
  File "/Users/bendichter/Development/pynwb/src/pynwb/form/backends/hdf5/h5tools.py", line 269, in write_builder
    self.__add_refs()
  File "/Users/bendichter/Development/pynwb/src/pynwb/form/backends/hdf5/h5tools.py", line 287, in __add_refs
    raise RuntimeError('Unable to resolve reference')
RuntimeError: Unable to resolve reference

bendichter commented 6 years ago

@ajtritt can you confirm that it's fixed in your version? Then I'll see if I can fix mine.

neuromusic commented 6 years ago

re-opening issue. I get the same error on f2054ac4ca2e01ba341c5a9c1fac06d2c41f1e3b

neuromusic commented 6 years ago

oh wait, I see.

it looks like @ajtritt's fix was to update the documentation, which is where the bug is. this is why your script is failing @bendichter.

when I update @bendichter's script with the changes that @ajtritt made at https://github.com/NeurodataWithoutBorders/pynwb/commit/9b6ebd31246cf8ea767f203d91d3a79cd51cbb41, I don't get any errors

bendichter commented 6 years ago

ah! I see that now. Thanks, @ajtritt and @neuromusic

NeurodataWithoutBorders / pynwb

demo code causes error: unresolved reference on 0.2.0 and on current dev branch #285

Steps to Reproduce

Environment