search

NeurodataWithoutBorders / pynwb

A Python API for working with Neurodata stored in the NWB Format
https://pynwb.readthedocs.io
Other
176 stars 84 forks source link

'Cannot reassign parent to Container' error with current hdmf version #982

Open luiztauffer opened 5 years ago

luiztauffer commented 5 years ago

1) Bug

With the current version of HDMF on github an error Cannot reassign parent to Container is raised if I try to set the same electrodes DynamicTableRegion to more than one module. The following code runs without errors on current PyPi hdmf 1.0.4 version, but with the github version, it breaks: (I wasn’t sure if it’s better to open this issues here or at hdmf, so sorry in advance if it should be put over there)

from datetime import datetime
from dateutil.tz import tzlocal
from pynwb import NWBFile, NWBHDF5IO, TimeSeries, ProcessingModule
from pynwb.ecephys import LFP, ElectricalSeries
import nwbext_ecog
import numpy as np
import os

# Creates file 1
nwb = NWBFile(session_description='session', identifier='1', session_start_time=datetime.now(tzlocal()))

# Add electrode groups and channels
nChannels = 10
dev0 = nwb.create_device(name='dev0')
elecs_group = nwb.create_electrode_group(name='electrodes', description='', location='ctx', device=dev0)
for i in np.arange(nChannels):
    ii = float(i)
    nwb.add_electrode(x=ii, y=ii, z=ii, imp=ii, location='', filtering='', group=elecs_group)

#Add signal
elecs_region = nwb.electrodes.create_region(name='electrodes',
                                            region=np.arange(nChannels).tolist(),
                                            description='')
X_data = np.zeros((nChannels,1000))
signal = ElectricalSeries(name='ElectricalSeries', 
                          data=X_data, 
                          electrodes=elecs_region,
                          rate=1.)
nwb.add_acquisition(signal)

# Extracellular group
ecephys_module = ProcessingModule(name='ecephys',
                                  description='Extracellular electrophysiology data.')
nwb.add_processing_module(ecephys_module)

# Create LFP data
lfp = LFP()
lfp_ts = lfp.create_electrical_series(name='preprocessed',
                                      data=X_data.T,
                                      electrodes=nwb.acquisition['ElectricalSeries'].electrodes,
                                      rate=1.,
                                      description='')
ecephys_module.add_data_interface(lfp)

#Write file
with NWBHDF5IO('file_1.nwb', mode='w') as io:
    io.write(nwb)

The error:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-1-e1d9ac22c674> in <module>
     40                                       electrodes=nwb.acquisition['ElectricalSeries'].electrodes,
     41                                       rate=1.,
---> 42                                       description='')
     43 ecephys_module.add_data_interface(lfp)
     44 

~/anaconda3/envs/ecog_vis/lib/python3.7/site-packages/hdmf/utils.py in func_call(*args, **kwargs)
    386                         raise_from(ExceptionType(msg), None)
    387 
--> 388                 return func(self, **parsed['args'])
    389         else:
    390             def func_call(*args, **kwargs):

~/anaconda3/envs/ecog_vis/lib/python3.7/site-packages/pynwb/core.py in _func(self, **kwargs)
    755         def _func(self, **kwargs):
    756             cargs, ckwargs = fmt_docval_args(container_type.__init__, kwargs)
--> 757             ret = container_type(*cargs, **ckwargs)
    758             getattr(self, add_name)(ret)
    759             return ret

~/anaconda3/envs/ecog_vis/lib/python3.7/site-packages/hdmf/utils.py in func_call(*args, **kwargs)
    386                         raise_from(ExceptionType(msg), None)
    387 
--> 388                 return func(self, **parsed['args'])
    389         else:
    390             def func_call(*args, **kwargs):

~/anaconda3/envs/ecog_vis/lib/python3.7/site-packages/pynwb/ecephys.py in __init__(self, **kwargs)
     94         name, electrodes, data = popargs('name', 'electrodes', 'data', kwargs)
     95         super(ElectricalSeries, self).__init__(name, data, 'volt', **kwargs)
---> 96         self.electrodes = electrodes
     97 
     98 

~/anaconda3/envs/ecog_vis/lib/python3.7/site-packages/pynwb/core.py in nwbdi_setter(self, val)
    266                             val = [val]
    267                         for v in val:
--> 268                             self.add_child(v)
    269 
    270                 ret.append(nwbdi_setter)

~/anaconda3/envs/ecog_vis/lib/python3.7/site-packages/hdmf/utils.py in func_call(*args, **kwargs)
    386                         raise_from(ExceptionType(msg), None)
    387 
--> 388                 return func(self, **parsed['args'])
    389         else:
    390             def func_call(*args, **kwargs):

~/anaconda3/envs/ecog_vis/lib/python3.7/site-packages/hdmf/container.py in add_child(self, **kwargs)
     52             self.__children.append(child)
     53             self.set_modified()
---> 54             child.parent = self
     55         else:
     56             warn('Cannot add None as child to a container %s' % self.name)

~/anaconda3/envs/ecog_vis/lib/python3.7/site-packages/hdmf/container.py in parent(self, parent_container)
     95             if isinstance(self.parent, Container):
     96                 raise ValueError(('Cannot reassign parent to Container: %s. '
---> 97                                   'Parent is already: %s.' % (repr(self), repr(self.parent))))
     98             else:
     99                 if parent_container is None:

ValueError: Cannot reassign parent to Container: 
electrodes <class 'pynwb.core.DynamicTableRegion'>
Fields:
  description: 
  table: electrodes <class 'pynwb.core.DynamicTable'>
. Parent is already: 
ElectricalSeries <class 'pynwb.ecephys.ElectricalSeries'>
Fields:
  comments: no comments
  conversion: 1.0
  data: [[0. 0. 0. ... 0. 0. 0.]
 [0. 0. 0. ... 0. 0. 0.]
 [0. 0. 0. ... 0. 0. 0.]
 ...
 [0. 0. 0. ... 0. 0. 0.]
 [0. 0. 0. ... 0. 0. 0.]
 [0. 0. 0. ... 0. 0. 0.]]
  description: no description
  electrodes: electrodes <class 'pynwb.core.DynamicTableRegion'>
  num_samples: 10
  rate: 1.0
  resolution: 0.0
  starting_time: 0.0
  unit: volt
rly commented 5 years ago

Thanks for the report. While we sort out what the right behavior should be here, as a workaround, you can redefine elecs_region before you use it the second time.

elecs_region = nwb.electrodes.create_region(name='electrodes',
                                            region=np.arange(nChannels).tolist(),
                                            description='')
lfp_ts = lfp.create_electrical_series(name='preprocessed',
                                      data=X_data.T,
                                      electrodes=elecs_region,
                                      rate=1.,
                                      description='')
luiztauffer commented 5 years ago

@rly thanks, that solves for this example. In the bigger project I'm working on, though, this workaround raises another error, at writing:

File "/home/tauffer/Github/ecogVIS/ecogvis/signal_processing/processing_data.py", line 224, in preprocess_raw_data
    io.write(nwb)
  File "/home/tauffer/anaconda3/envs/ecog_vis/lib/python3.7/site-packages/hdmf/utils.py", line 388, in func_call
    return func(self, **parsed['args'])
  File "/home/tauffer/anaconda3/envs/ecog_vis/lib/python3.7/site-packages/hdmf/backends/hdf5/h5tools.py", line 219, in write
    call_docval_func(super(HDF5IO, self).write, kwargs)
  File "/home/tauffer/anaconda3/envs/ecog_vis/lib/python3.7/site-packages/hdmf/utils.py", line 281, in call_docval_func
    return func(*fargs, **fkwargs)
  File "/home/tauffer/anaconda3/envs/ecog_vis/lib/python3.7/site-packages/hdmf/utils.py", line 388, in func_call
    return func(self, **parsed['args'])
  File "/home/tauffer/anaconda3/envs/ecog_vis/lib/python3.7/site-packages/hdmf/backends/io.py", line 42, in write
    self.write_builder(f_builder, **kwargs)
  File "/home/tauffer/anaconda3/envs/ecog_vis/lib/python3.7/site-packages/hdmf/utils.py", line 388, in func_call
    return func(self, **parsed['args'])
  File "/home/tauffer/anaconda3/envs/ecog_vis/lib/python3.7/site-packages/hdmf/backends/hdf5/h5tools.py", line 446, in write_builder
    self.write_group(self.__file, gbldr)
  File "/home/tauffer/anaconda3/envs/ecog_vis/lib/python3.7/site-packages/hdmf/utils.py", line 388, in func_call
    return func(self, **parsed['args'])
  File "/home/tauffer/anaconda3/envs/ecog_vis/lib/python3.7/site-packages/hdmf/backends/hdf5/h5tools.py", line 599, in write_group
    self.write_link(group, sub_builder)
  File "/home/tauffer/anaconda3/envs/ecog_vis/lib/python3.7/site-packages/hdmf/utils.py", line 388, in func_call
    return func(self, **parsed['args'])
  File "/home/tauffer/anaconda3/envs/ecog_vis/lib/python3.7/site-packages/hdmf/backends/hdf5/h5tools.py", line 636, in write_link
    parent[name] = link_obj
  File "/home/tauffer/anaconda3/envs/ecog_vis/lib/python3.7/site-packages/h5py/_hl/group.py", line 392, in __setitem__
    lcpl=lcpl, lapl=self._lapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5l.pyx", line 157, in h5py.h5l.LinkProxy.create_external
RuntimeError: Unable to create link (name already exists)

I still couldn't isolate the cause of it in order to give a simple reproducible code, but maybe you get an idea of what might be? I'll update it here if I can figure it out. Importantly, this only shows up with the github installation of hdmf, the version on pypi runs normally

rly commented 5 years ago

Oh. It might be because you have two electrode table regions with the same name, and that is (currently) not allowed.

On Tue, Jun 25, 2019 at 10:38 AM Luiz Tauffer notifications@github.com wrote:

@rly https://github.com/rly thanks, that solves for this example. In the bigger project I'm working on, though, this workaround raises another error:

File "/home/tauffer/Github/ecogVIS/ecogvis/signal_processing/processing_data.py", line 224, in preprocess_raw_data io.write(nwb) File "/home/tauffer/anaconda3/envs/ecog_vis/lib/python3.7/site-packages/hdmf/utils.py", line 388, in func_call return func(self, *parsed['args']) File "/home/tauffer/anaconda3/envs/ecog_vis/lib/python3.7/site-packages/hdmf/backends/hdf5/h5tools.py", line 219, in write call_docval_func(super(HDF5IO, self).write, kwargs) File "/home/tauffer/anaconda3/envs/ecog_vis/lib/python3.7/site-packages/hdmf/utils.py", line 281, in call_docval_func return func(fargs, fkwargs) File "/home/tauffer/anaconda3/envs/ecog_vis/lib/python3.7/site-packages/hdmf/utils.py", line 388, in func_call return func(self, parsed['args']) File "/home/tauffer/anaconda3/envs/ecog_vis/lib/python3.7/site-packages/hdmf/backends/io.py", line 42, in write self.write_builder(f_builder, kwargs) File "/home/tauffer/anaconda3/envs/ecog_vis/lib/python3.7/site-packages/hdmf/utils.py", line 388, in func_call return func(self, parsed['args']) File "/home/tauffer/anaconda3/envs/ecog_vis/lib/python3.7/site-packages/hdmf/backends/hdf5/h5tools.py", line 446, in write_builder self.write_group(self.__file, gbldr) File "/home/tauffer/anaconda3/envs/ecog_vis/lib/python3.7/site-packages/hdmf/utils.py", line 388, in func_call return func(self, parsed['args']) File "/home/tauffer/anaconda3/envs/ecog_vis/lib/python3.7/site-packages/hdmf/backends/hdf5/h5tools.py", line 599, in write_group self.write_link(group, sub_builder) File "/home/tauffer/anaconda3/envs/ecog_vis/lib/python3.7/site-packages/hdmf/utils.py", line 388, in func_call return func(self, parsed['args']) File "/home/tauffer/anaconda3/envs/ecog_vis/lib/python3.7/site-packages/hdmf/backends/hdf5/h5tools.py", line 636, in write_link parent[name] = link_obj File "/home/tauffer/anaconda3/envs/ecog_vis/lib/python3.7/site-packages/h5py/_hl/group.py", line 392, in setitem lcpl=lcpl, lapl=self._lapl) File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper File "h5py/h5l.pyx", line 157, in h5py.h5l.LinkProxy.create_externalRuntimeError: Unable to create link (name already exists)

I still couldn't isolate the cause of it in order to give a simple reproducible code, but maybe you get an idea of what might be? I'll update it here if I can figure it out. Importantly, this only shows up with the github installation of hdmf, the version on pypi runs normally

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/NeurodataWithoutBorders/pynwb/issues/982?email_source=notifications&email_token=AACLXNO3W5DEO4JFW6MFJD3P4JJZPA5CNFSM4H3KAVVKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYRATOA#issuecomment-505547192, or mute the thread https://github.com/notifications/unsubscribe-auth/AACLXNIIMOCS3OIKDC3CNPLP4JJZPANCNFSM4H3KAVVA .

-- Ryan Ly Scientific Data Engineer Lawrence Berkeley National Laboratory 1 Cyclotron Road Mail Stop 59R4104 Berkeley, CA 94720 Email: rly@lbl.gov Pronouns: he/him

luiztauffer commented 5 years ago

Ok, so the problem seems to be incompatibility between NWB files created with previous versions of hdmf and the new one (github version). I created a new NWB file with the same data as my old file (older hdmf) and this new file can be written just fine now.

rly commented 5 years ago

I've updated hdmf to 1.0.5. That should fix the problem. Please re-open and comment if it does not.

luiztauffer commented 5 years ago

@rly it dd not solve the problem here, getting the same error as before

rly commented 5 years ago

@luiztauffer sorry, can you clarify the current problem? Are you still getting a RuntimeError: Unable to create link (name already exists) ? I am trying to mock together something that reproduces your error based on what I see in https://github.com/luiztauffer/ecogVIS/blob/master/ecogvis/signal_processing/processing_data.py but have so far not been successful.