search

hdmf-dev / hdmf

The Hierarchical Data Modeling Framework
http://hdmf.readthedocs.io
Other
46 stars 26 forks source link

Write fails when refining target_type of object reference attribute #289

Open oruebel opened 4 years ago

oruebel commented 4 years ago

Description

In the icephys extension we overwrite the target type for the DynamicTableRegion to point to a specific type of DynamicTable, e.g., here as follows:

- name: recordings
    neurodata_type_inc: DynamicTableRegion
    doc: A reference to one or more rows in the IntracellularRecordings table.
    attributes:
    - name: table
      dtype:
        target_type: IntracellularRecordings
        reftype: object

This used to work but now fails at line:

https://github.com/hdmf-dev/hdmf/blob/2d467c0a4d22f0a4fa373152a2c6145e1440efbb/src/hdmf/build/objectmapper.py#L591

with the error message shown below. It looks like #279 added a check for overwritten attributes that now fails on the extension. If I change the line to

all_attrs = self.__spec.attributes # + getattr(spec_ext, 'attributes', tuple())

then the write works, but its not the right fix since it means we do not actually check the attribute.

Here the error message.

  File "example.py", line 64, in <module>
    io.write(nwbfile)
  File "/Users/oruebel/Devel/nwb/hdmf/src/hdmf/utils.py", line 453, in func_call
    return func(args[0], **parsed['args'])
  File "/Users/oruebel/Devel/nwb/hdmf/src/hdmf/backends/hdf5/h5tools.py", line 269, in write
    call_docval_func(super().write, kwargs)
  File "/Users/oruebel/Devel/nwb/hdmf/src/hdmf/utils.py", line 343, in call_docval_func
    return func(*fargs, **fkwargs)
  File "/Users/oruebel/Devel/nwb/hdmf/src/hdmf/utils.py", line 453, in func_call
    return func(args[0], **parsed['args'])
  File "/Users/oruebel/Devel/nwb/hdmf/src/hdmf/backends/io.py", line 43, in write
    f_builder = self.__manager.build(container, source=self.__source)
  File "/Users/oruebel/Devel/nwb/hdmf/src/hdmf/utils.py", line 453, in func_call
    return func(args[0], **parsed['args'])
  File "/Users/oruebel/Devel/nwb/hdmf/src/hdmf/build/manager.py", line 156, in build
    result = self.__type_map.build(container, self, source=source, spec_ext=spec_ext)
  File "/Users/oruebel/Devel/nwb/hdmf/src/hdmf/utils.py", line 453, in func_call
    return func(args[0], **parsed['args'])
  File "/Users/oruebel/Devel/nwb/hdmf/src/hdmf/build/manager.py", line 743, in build
    builder = obj_mapper.build(container, manager, builder=builder, source=source, spec_ext=spec_ext)
  File "/Users/oruebel/Devel/nwb/hdmf/src/hdmf/utils.py", line 453, in func_call
    return func(args[0], **parsed['args'])
  File "/Users/oruebel/Devel/nwb/hdmf/src/hdmf/build/objectmapper.py", line 539, in build
    self.__add_groups(builder, self.__spec.groups, container, manager, source)
  File "/Users/oruebel/Devel/nwb/hdmf/src/hdmf/build/objectmapper.py", line 756, in __add_groups
    self.__add_groups(sub_builder, spec.groups, container, build_manager, source)
  File "/Users/oruebel/Devel/nwb/hdmf/src/hdmf/build/objectmapper.py", line 756, in __add_groups
    self.__add_groups(sub_builder, spec.groups, container, build_manager, source)
  File "/Users/oruebel/Devel/nwb/hdmf/src/hdmf/build/objectmapper.py", line 772, in __add_groups
    self.__add_containers(builder, spec, attr_value, build_manager, source, container)
  File "/Users/oruebel/Devel/nwb/hdmf/src/hdmf/build/objectmapper.py", line 783, in __add_containers
    rendered_obj = build_manager.build(value, source=source, spec_ext=spec)
  File "/Users/oruebel/Devel/nwb/hdmf/src/hdmf/utils.py", line 453, in func_call
    return func(args[0], **parsed['args'])
  File "/Users/oruebel/Devel/nwb/hdmf/src/hdmf/build/manager.py", line 156, in build
    result = self.__type_map.build(container, self, source=source, spec_ext=spec_ext)
  File "/Users/oruebel/Devel/nwb/hdmf/src/hdmf/utils.py", line 453, in func_call
    return func(args[0], **parsed['args'])
  File "/Users/oruebel/Devel/nwb/hdmf/src/hdmf/build/manager.py", line 743, in build
    builder = obj_mapper.build(container, manager, builder=builder, source=source, spec_ext=spec_ext)
  File "/Users/oruebel/Devel/nwb/hdmf/src/hdmf/utils.py", line 453, in func_call
    return func(args[0], **parsed['args'])
  File "/Users/oruebel/Devel/nwb/hdmf/src/hdmf/build/objectmapper.py", line 538, in build
    self.__add_datasets(builder, self.__spec.datasets, container, manager, source)
  File "/Users/oruebel/Devel/nwb/hdmf/src/hdmf/build/objectmapper.py", line 732, in __add_datasets
    self.__add_containers(builder, spec, attr_value, build_manager, source, container)
  File "/Users/oruebel/Devel/nwb/hdmf/src/hdmf/build/objectmapper.py", line 783, in __add_containers
    rendered_obj = build_manager.build(value, source=source, spec_ext=spec)
  File "/Users/oruebel/Devel/nwb/hdmf/src/hdmf/utils.py", line 453, in func_call
    return func(args[0], **parsed['args'])
  File "/Users/oruebel/Devel/nwb/hdmf/src/hdmf/build/manager.py", line 156, in build
    result = self.__type_map.build(container, self, source=source, spec_ext=spec_ext)
  File "/Users/oruebel/Devel/nwb/hdmf/src/hdmf/utils.py", line 453, in func_call
    return func(args[0], **parsed['args'])
  File "/Users/oruebel/Devel/nwb/hdmf/src/hdmf/build/manager.py", line 743, in build
    builder = obj_mapper.build(container, manager, builder=builder, source=source, spec_ext=spec_ext)
  File "/Users/oruebel/Devel/nwb/hdmf/src/hdmf/utils.py", line 453, in func_call
    return func(args[0], **parsed['args'])
  File "/Users/oruebel/Devel/nwb/hdmf/src/hdmf/build/objectmapper.py", line 592, in build
    self.__add_attributes(builder, all_attrs, container, manager, source)
  File "/Users/oruebel/Devel/nwb/hdmf/src/hdmf/build/objectmapper.py", line 681, in __add_attributes
    raise ValueError(msg)
ValueError: object of data_type IntracellularRecordings not found on DynamicTableRegion 'recordings'

Here an example to reproduce the problem:

from datetime import datetime
from dateutil.tz import tzlocal
import numpy as np
from pynwb.icephys import VoltageClampStimulusSeries, VoltageClampSeries
from pynwb import NWBHDF5IO
from ndx_icephys_meta.icephys import ICEphysFile

nwbfile = ICEphysFile(session_description='my first recording',
                      identifier='EXAMPLE_ID',
                      session_start_time=datetime.now(tzlocal()))

# Add a device
device = nwbfile.create_device(name='Heka ITC-1600')

# Add an intracellular electrode
electrode = nwbfile.create_ic_electrode(name="elec0",
                                        description='a mock intracellular electrode',
                                        device=device)

# Create an ic-ephys stimulus
stimulus = VoltageClampStimulusSeries(
            name="stimulus",
            data=[1, 2, 3, 4, 5],
            starting_time=123.6,
            rate=10e3,
            electrode=electrode,
            gain=0.02,
            sweep_number=15)

# Create an ic-response
response = VoltageClampSeries(
            name='response',
            data=[0.1, 0.2, 0.3, 0.4, 0.5],
            conversion=1e-12,
            resolution=np.nan,
            starting_time=123.6,
            rate=20e3,
            electrode=electrode,
            gain=0.02,
            capacitance_slow=100e-12,
            resistance_comp_correction=70.0,
            sweep_number=15)

# Add an intracellular recording to the file
nwbfile.add_intracellular_recording(electrode=electrode,
                                    stimulus=stimulus,
                                    response=response)

# Add a list of sweeps to the sweeps table
nwbfile.add_ic_sweep(recordings=[0])

# Write our test file
testpath = "test_icephys_file.h5"
with NWBHDF5IO(testpath, 'w') as io:
    io.write(nwbfile)

Checklist

oruebel commented 4 years ago

@ajtritt I think I'll need your help with this one

rly commented 3 years ago

This issue appears to be resolved now. The minimum working example (after updating to run in the latest HDMF) no longer generates the error.

Before closing, we should add a test for refining target_type of an object reference attribute to make sure.