NeurodataWithoutBorders / pynwb

A Python API for working with Neurodata stored in the NWB Format
https://pynwb.readthedocs.io
Other
177 stars 84 forks source link

[Bug]: failure to load data with ndx-ecog extension #1613

Open Christonikos opened 1 year ago

Christonikos commented 1 year ago

What happened?

When trying to load an nwb file that contains ECOG data, I receive a ConstructError originating from a TypeError: CustomClassGenerator.set_init.<locals>.__init__: missing argument 'parcellations.

Notably, the file could be previously loaded on a dell, ubuntu-based laptop, and the problem occurs when trying to load the file on an apple m1 macbook air.

The code originally used to load the file was:

 from pynwb import NWBHDF5IO

 io = NWBHDF5IO(fname, 'r', load_namespaces=True)
 nwb2 = io.read()

After interacting a bit with @bendichter I tried his suggestion (that works on his m1):

import pynwb
import h5py

with h5py.File(fname) as file:
    with pynwb.NWBHDF5IO(file=file, load_namespaces=True) as io:
        nwbfile = io.read()
        print(nwbfile)

Which led to the same exact error on my side.

I am bit puzzled by this, so I opened it as an issue in the hopes that this can also be useful for the members of the community.

I am using pynwb #2.2.0 (but also tried with many other versions) and h5py #2.10.0 Screenshot 2022-12-26 at 11 06 52 Screenshot 2022-12-26 at 11 06 32

Steps to Reproduce

# load the .mat file
        fname = '..._ieeg.nwb'

        io = NWBHDF5IO(fname, 'r', load_namespaces=True)
        nwb2 = io.read()

with h5py.File(fname) as file:
    with pynwb.NWBHDF5IO(file=file, load_namespaces=True) as io:
        nwbfile = io.read()
        print(nwbfile)

Traceback

Traceback (most recent call last):

  File ~/anaconda3/lib/python3.9/site-packages/hdmf/build/objectmapper.py:1248 in construct
    obj = self.__new_container__(cls, builder.source, parent, builder.attributes.get(self.__spec.id_key()),

  File ~/anaconda3/lib/python3.9/site-packages/hdmf/build/objectmapper.py:1261 in __new_container__
    obj.__init__(**kwargs)

  File ~/anaconda3/lib/python3.9/site-packages/hdmf/utils.py:645 in func_call
    return func(args[0], **pargs)

  File ~/anaconda3/lib/python3.9/site-packages/hdmf/build/classgenerator.py:411 in __init__
    previous_init(self, **kwargs)

  File ~/anaconda3/lib/python3.9/site-packages/hdmf/utils.py:644 in func_call
    pargs = _check_args(args, kwargs)

  File ~/anaconda3/lib/python3.9/site-packages/hdmf/utils.py:637 in _check_args
    raise ExceptionType(msg)

TypeError: CustomClassGenerator.set_init.<locals>.__init__: missing argument 'parcellations'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):

  Input In [13] in <cell line: 1>
    nwbfile = io.read()

  File ~/anaconda3/lib/python3.9/site-packages/hdmf/backends/hdf5/h5tools.py:451 in read
    return super().read(**kwargs)

  File ~/anaconda3/lib/python3.9/site-packages/hdmf/utils.py:645 in func_call
    return func(args[0], **pargs)

  File ~/anaconda3/lib/python3.9/site-packages/hdmf/backends/io.py:42 in read
    container = self.__manager.construct(f_builder)

  File ~/anaconda3/lib/python3.9/site-packages/hdmf/utils.py:645 in func_call
    return func(args[0], **pargs)

  File ~/anaconda3/lib/python3.9/site-packages/hdmf/build/manager.py:280 in construct
    result = self.__type_map.construct(builder, self, None)

  File ~/anaconda3/lib/python3.9/site-packages/hdmf/utils.py:645 in func_call
    return func(args[0], **pargs)

  File ~/anaconda3/lib/python3.9/site-packages/hdmf/build/manager.py:789 in construct
    return obj_mapper.construct(builder, build_manager, parent)

  File ~/anaconda3/lib/python3.9/site-packages/hdmf/utils.py:645 in func_call
    return func(args[0], **pargs)

  File ~/anaconda3/lib/python3.9/site-packages/hdmf/build/objectmapper.py:1218 in construct
    subspecs = self.__get_subspec_values(builder, self.spec, manager)

  File ~/anaconda3/lib/python3.9/site-packages/hdmf/build/objectmapper.py:1147 in __get_subspec_values
    self.__get_sub_builders(groups, spec.groups, manager, ret)

  File ~/anaconda3/lib/python3.9/site-packages/hdmf/build/objectmapper.py:1198 in __get_sub_builders
    ret.update(self.__get_subspec_values(sub_builder, subspec, manager))

  File ~/anaconda3/lib/python3.9/site-packages/hdmf/build/objectmapper.py:1147 in __get_subspec_values
    self.__get_sub_builders(groups, spec.groups, manager, ret)

  File ~/anaconda3/lib/python3.9/site-packages/hdmf/build/objectmapper.py:1200 in __get_sub_builders
    ret[subspec] = manager.construct(sub_builder)

  File ~/anaconda3/lib/python3.9/site-packages/hdmf/utils.py:645 in func_call
    return func(args[0], **pargs)

  File ~/anaconda3/lib/python3.9/site-packages/hdmf/build/manager.py:276 in construct
    result = self.__type_map.construct(builder, self, parent)

  File ~/anaconda3/lib/python3.9/site-packages/hdmf/utils.py:645 in func_call
    return func(args[0], **pargs)

  File ~/anaconda3/lib/python3.9/site-packages/hdmf/build/manager.py:789 in construct
    return obj_mapper.construct(builder, build_manager, parent)

  File ~/anaconda3/lib/python3.9/site-packages/hdmf/utils.py:645 in func_call
    return func(args[0], **pargs)

  File ~/anaconda3/lib/python3.9/site-packages/hdmf/build/objectmapper.py:1218 in construct
    subspecs = self.__get_subspec_values(builder, self.spec, manager)

  File ~/anaconda3/lib/python3.9/site-packages/hdmf/build/objectmapper.py:1147 in __get_subspec_values
    self.__get_sub_builders(groups, spec.groups, manager, ret)

  File ~/anaconda3/lib/python3.9/site-packages/hdmf/build/objectmapper.py:1190 in __get_sub_builders
    sub_builder = self.__flatten(sub_builder, subspec, manager)

  File ~/anaconda3/lib/python3.9/site-packages/hdmf/build/objectmapper.py:1203 in __flatten
    tmp = [manager.construct(b) for b in sub_builder]

  File ~/anaconda3/lib/python3.9/site-packages/hdmf/build/objectmapper.py:1203 in <listcomp>
    tmp = [manager.construct(b) for b in sub_builder]

  File ~/anaconda3/lib/python3.9/site-packages/hdmf/utils.py:645 in func_call
    return func(args[0], **pargs)

  File ~/anaconda3/lib/python3.9/site-packages/hdmf/build/manager.py:276 in construct
    result = self.__type_map.construct(builder, self, parent)

  File ~/anaconda3/lib/python3.9/site-packages/hdmf/utils.py:645 in func_call
    return func(args[0], **pargs)

  File ~/anaconda3/lib/python3.9/site-packages/hdmf/build/manager.py:789 in construct
    return obj_mapper.construct(builder, build_manager, parent)

  File ~/anaconda3/lib/python3.9/site-packages/hdmf/utils.py:645 in func_call
    return func(args[0], **pargs)

  File ~/anaconda3/lib/python3.9/site-packages/hdmf/build/objectmapper.py:1218 in construct
    subspecs = self.__get_subspec_values(builder, self.spec, manager)

  File ~/anaconda3/lib/python3.9/site-packages/hdmf/build/objectmapper.py:1147 in __get_subspec_values
    self.__get_sub_builders(groups, spec.groups, manager, ret)

  File ~/anaconda3/lib/python3.9/site-packages/hdmf/build/objectmapper.py:1190 in __get_sub_builders
    sub_builder = self.__flatten(sub_builder, subspec, manager)

  File ~/anaconda3/lib/python3.9/site-packages/hdmf/build/objectmapper.py:1203 in __flatten
    tmp = [manager.construct(b) for b in sub_builder]

  File ~/anaconda3/lib/python3.9/site-packages/hdmf/build/objectmapper.py:1203 in <listcomp>
    tmp = [manager.construct(b) for b in sub_builder]

  File ~/anaconda3/lib/python3.9/site-packages/hdmf/utils.py:645 in func_call
    return func(args[0], **pargs)

  File ~/anaconda3/lib/python3.9/site-packages/hdmf/build/manager.py:276 in construct
    result = self.__type_map.construct(builder, self, parent)

  File ~/anaconda3/lib/python3.9/site-packages/hdmf/utils.py:645 in func_call
    return func(args[0], **pargs)

  File ~/anaconda3/lib/python3.9/site-packages/hdmf/build/manager.py:789 in construct
    return obj_mapper.construct(builder, build_manager, parent)

  File ~/anaconda3/lib/python3.9/site-packages/hdmf/utils.py:645 in func_call
    return func(args[0], **pargs)

  File ~/anaconda3/lib/python3.9/site-packages/hdmf/build/objectmapper.py:1218 in construct
    subspecs = self.__get_subspec_values(builder, self.spec, manager)

  File ~/anaconda3/lib/python3.9/site-packages/hdmf/build/objectmapper.py:1147 in __get_subspec_values
    self.__get_sub_builders(groups, spec.groups, manager, ret)

  File ~/anaconda3/lib/python3.9/site-packages/hdmf/build/objectmapper.py:1190 in __get_sub_builders
    sub_builder = self.__flatten(sub_builder, subspec, manager)

  File ~/anaconda3/lib/python3.9/site-packages/hdmf/build/objectmapper.py:1203 in __flatten
    tmp = [manager.construct(b) for b in sub_builder]

  File ~/anaconda3/lib/python3.9/site-packages/hdmf/build/objectmapper.py:1203 in <listcomp>
    tmp = [manager.construct(b) for b in sub_builder]

  File ~/anaconda3/lib/python3.9/site-packages/hdmf/utils.py:645 in func_call
    return func(args[0], **pargs)

  File ~/anaconda3/lib/python3.9/site-packages/hdmf/build/manager.py:276 in construct
    result = self.__type_map.construct(builder, self, parent)

  File ~/anaconda3/lib/python3.9/site-packages/hdmf/utils.py:645 in func_call
    return func(args[0], **pargs)

  File ~/anaconda3/lib/python3.9/site-packages/hdmf/build/manager.py:789 in construct
    return obj_mapper.construct(builder, build_manager, parent)

  File ~/anaconda3/lib/python3.9/site-packages/hdmf/utils.py:645 in func_call
    return func(args[0], **pargs)

  File ~/anaconda3/lib/python3.9/site-packages/hdmf/build/objectmapper.py:1252 in construct
    raise ConstructError(builder, msg) from ex

ConstructError: (root/general/subject/cortical_surfaces/std_inflated_LH/parcellations GroupBuilder {'attributes': {'namespace': 'ndx-ecog', 'neurodata_type': 'Parcellations', 'object_id': '1d2c5f0a-9772-4867-afb5-e2ad1ccb6248'}, 'groups': {}, 'datasets': {'HCP': root/general/subject/cortical_surfaces/std_inflated_LH/parcellations/HCP DatasetBuilder {'attributes': {'labels': array([b'???', b'V1', b'MST', b'V6', b'V2', b'V3', b'V4', b'V8', b'4',
       b'3b', b'FEF', b'PEF', b'55b', b'V3A', b'RSC', b'POS2', b'V7',
       b'IPS1', b'FFC', b'V3B', b'LO1', b'LO2', b'PIT', b'MT', b'A1',
       b'PSL', b'SFL', b'PCV', b'STV', b'7Pm', b'7m', b'POS1', b'23d',
       b'v23ab', b'd23ab', b'31pv', b'5m', b'5mv', b'23c', b'5L', b'24dd',
       b'24dv', b'7AL', b'SCEF', b'6ma', b'7Am', b'7PL', b'7PC', b'LIPv',
       b'VIP', b'MIP', b'1', b'2', b'3a', b'6d', b'6mp', b'6v', b'p24pr',
       b'33pr', b'a24pr', b'p32pr', b'a24', b'd32', b'8BM', b'p32',
       b'10r', b'47m', b'8Av', b'8Ad', b'9m', b'8BL', b'9p', b'10d',
       b'8C', b'44', b'45', b'47l', b'a47r', b'6r', b'IFJa', b'IFJp',
       b'IFSp', b'IFSa', b'p9-46v', b'46', b'a9-46v', b'9-46d', b'9a',
       b'10v', b'a10p', b'10pp', b'11l', b'13l', b'OFC', b'47s', b'LIPd',
       b'6a', b'i6-8', b's6-8', b'43', b'OP4', b'OP1', b'OP2-3', b'52',
       b'RI', b'PFcm', b'PoI2', b'TA2', b'FOP4', b'MI', b'Pir', b'AVI',
       b'AAIC', b'FOP1', b'FOP3', b'FOP2', b'PFt', b'AIP', b'EC', b'PreS',
       b'H', b'ProS', b'PeEc', b'STGa', b'PBelt', b'A5', b'PHA1', b'PHA3',
       b'STSda', b'STSdp', b'STSvp', b'TGd', b'TE1a', b'TE1p', b'TE2a',
       b'TF', b'TE2p', b'PHT', b'PH', b'TPOJ1', b'TPOJ2', b'TPOJ3',
       b'DVT', b'PGp', b'IP2', b'IP1', b'IP0', b'PFop', b'PF', b'PFm',
       b'PGi', b'PGs', b'V6A', b'VMV1', b'VMV3', b'PHA2', b'V4t', b'FST',
       b'V3CD', b'LO3', b'VMV2', b'31pd', b'31a', b'VVC', b'25', b's32',
       b'pOFC', b'PoI1', b'Ig', b'FOP5', b'p10p', b'p47r', b'TGv',
       b'MBelt', b'LBelt', b'A4', b'STSva', b'TE1m', b'PI', b'a32pr',
       b'p24'], dtype=object), 'namespace': 'ndx-ecog', 'neurodata_type': 'Parcellation', 'object_id': '1a61a125-9ad7-47f3-a86c-0748cb25f835'}, 'data': <Closed HDF5 dataset>}, 'curv': root/general/subject/cortical_surfaces/std_inflated_LH/parcellations/curv DatasetBuilder {'attributes': {'namespace': 'ndx-ecog', 'neurodata_type': 'Parcellation', 'object_id': '38e414e1-7aa8-42e7-9ee8-08624621b7dd'}, 'data': <Closed HDF5 dataset>}}, 'links': {}}, "Could not construct Parcellations object due to: CustomClassGenerator.set_init.<locals>.__init__: missing argument 'parcellations'")

Operating System

macOS

Python Executable

Conda

Python Version

3.9

Package Versions

pynwb #2.2.0 h5py #2.10.0

Code of Conduct

rly commented 1 year ago

Hi @Christonikos , thanks for the bug report. Could you share the file that produces the error? If it is too big or sensitive to share, I can try to troubleshoot without it, but since the error appears to be due to the usage of an extension data type, it would be considerably easier to troubleshoot with the file.

Christonikos commented 1 year ago

Hi @rly thanks for the follow up. Please allow me to ask the hosting lab and I will get back to you.

bendichter commented 1 year ago

I really don't think M1 is the issue here. There are certain types of installation snafus with the new M1 macs, but this doesn't hit any of them as far as I can see.

I parcellations was added later in the ndx-ecog extension. My guess is there is a misalignment between the schema used to write the data and the schema that is being cached in the file. The data contains parcellations but the cached schema is older and does not.

bendichter commented 1 year ago

One quick fix might be to update ndx-ecog and import it before reading the file (or if you are already doing that, try not importing it)

Christonikos commented 1 year ago

I just tried with both ndx_ecog #0.1.1 and #0.1.0 and adding the importation prior to reading the file, but that led to the same error. My current pynwb==2.2.0.

Christonikos commented 1 year ago

I am attaching a file for troubleshooting here.

bendichter commented 1 year ago

I get the same error on my computer. It looks like the cached ndx-ecog schema is outdated. When I try importing it, I also get a traceback but it is different:

import pynwb
import ndx_ecog

filepath = "/Users/bendichter/Downloads/sub-TS106_places_ieeg_2.nwb"

with pynwb.NWBHDF5IO(filepath, load_namespaces=True) as io:
    nwbfile = io.read()
    print(nwbfile)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
File ~/opt/miniconda3/lib/python3.9/site-packages/hdmf/build/objectmapper.py:1248, in ObjectMapper.construct(self, **kwargs)
   1247 try:
-> 1248     obj = self.__new_container__(cls, builder.source, parent, builder.attributes.get(self.__spec.id_key()),
   1249                                  **kwargs)
   1250 except Exception as ex:

File ~/opt/miniconda3/lib/python3.9/site-packages/hdmf/build/objectmapper.py:1261, in ObjectMapper.__new_container__(self, cls, container_source, parent, object_id, **kwargs)
   1259 # obj has been created and is in construction mode, indicating that the object is being constructed by
   1260 # the automatic construct process during read, rather than by the user
-> 1261 obj.__init__(**kwargs)
   1262 obj._in_construct_mode = False  # reset to False to indicate that the construction of the object is complete

File ~/opt/miniconda3/lib/python3.9/site-packages/hdmf/utils.py:645, in docval.<locals>.dec.<locals>.func_call(*args, **kwargs)
    644 pargs = _check_args(args, kwargs)
--> 645 return func(args[0], **pargs)

File ~/opt/miniconda3/lib/python3.9/site-packages/ndx_ecog/ecog.py:24, in Surface.__init__(self, **kwargs)
     23 if np.max(faces) >= len(vertices):
---> 24     raise ValueError('index of faces exceeds number vertices for {}. '
     25                      'Faces should be 0-indexed, not 1-indexed'.
     26                      format(name))
     27 if np.min(faces) < 0:

ValueError: index of faces exceeds number vertices for std_inflated_LH. Faces should be 0-indexed, not 1-indexed

The above exception was the direct cause of the following exception:

ConstructError                            Traceback (most recent call last)
Cell In [2], line 7
      4 filepath = "/Users/bendichter/Downloads/sub-TS106_places_ieeg_2.nwb"
      6 with pynwb.NWBHDF5IO(filepath, load_namespaces=True) as io:
----> 7     nwbfile = io.read()
      8     print(nwbfile)

File ~/opt/miniconda3/lib/python3.9/site-packages/hdmf/backends/hdf5/h5tools.py:451, in HDF5IO.read(self, **kwargs)
    448     raise UnsupportedOperation("Cannot read from file %s in mode '%s'. Please use mode 'r', 'r+', or 'a'."
    449                                % (self.source, self.__mode))
    450 try:
--> 451     return super().read(**kwargs)
    452 except UnsupportedOperation as e:
    453     if str(e) == 'Cannot build data. There are no values.':  # pragma: no cover

File ~/opt/miniconda3/lib/python3.9/site-packages/hdmf/utils.py:645, in docval.<locals>.dec.<locals>.func_call(*args, **kwargs)
    643 def func_call(*args, **kwargs):
    644     pargs = _check_args(args, kwargs)
--> 645     return func(args[0], **pargs)

File ~/opt/miniconda3/lib/python3.9/site-packages/hdmf/backends/io.py:42, in HDMFIO.read(self, **kwargs)
     39 if all(len(v) == 0 for v in f_builder.values()):
     40     # TODO also check that the keys are appropriate. print a better error message
     41     raise UnsupportedOperation('Cannot build data. There are no values.')
---> 42 container = self.__manager.construct(f_builder)
     43 return container

File ~/opt/miniconda3/lib/python3.9/site-packages/hdmf/utils.py:645, in docval.<locals>.dec.<locals>.func_call(*args, **kwargs)
    643 def func_call(*args, **kwargs):
    644     pargs = _check_args(args, kwargs)
--> 645     return func(args[0], **pargs)

File ~/opt/miniconda3/lib/python3.9/site-packages/hdmf/build/manager.py:280, in BuildManager.construct(self, **kwargs)
    276     result = self.__type_map.construct(builder, self, parent)
    277 else:
    278     # we are at the top of the hierarchy,
    279     # so it must be time to resolve parents
--> 280     result = self.__type_map.construct(builder, self, None)
    281     self.__resolve_parents(result)
    282 self.prebuilt(result, builder)

File ~/opt/miniconda3/lib/python3.9/site-packages/hdmf/utils.py:645, in docval.<locals>.dec.<locals>.func_call(*args, **kwargs)
    643 def func_call(*args, **kwargs):
    644     pargs = _check_args(args, kwargs)
--> 645     return func(args[0], **pargs)

File ~/opt/miniconda3/lib/python3.9/site-packages/hdmf/build/manager.py:789, in TypeMap.construct(self, **kwargs)
    787     raise ValueError('No ObjectMapper found for builder of type %s' % dt)
    788 else:
--> 789     return obj_mapper.construct(builder, build_manager, parent)

File ~/opt/miniconda3/lib/python3.9/site-packages/hdmf/utils.py:645, in docval.<locals>.dec.<locals>.func_call(*args, **kwargs)
    643 def func_call(*args, **kwargs):
    644     pargs = _check_args(args, kwargs)
--> 645     return func(args[0], **pargs)

File ~/opt/miniconda3/lib/python3.9/site-packages/hdmf/build/objectmapper.py:1218, in ObjectMapper.construct(self, **kwargs)
   1216 cls = manager.get_cls(builder)
   1217 # gather all subspecs
-> 1218 subspecs = self.__get_subspec_values(builder, self.spec, manager)
   1219 # get the constructor argument that each specification corresponds to
   1220 const_args = dict()

File ~/opt/miniconda3/lib/python3.9/site-packages/hdmf/build/objectmapper.py:1147, in ObjectMapper.__get_subspec_values(self, builder, spec, manager)
   1145                 ret[subspec] = self.__flatten(sub_builder, subspec, manager)
   1146     # now process groups and datasets
-> 1147     self.__get_sub_builders(groups, spec.groups, manager, ret)
   1148     self.__get_sub_builders(datasets, spec.datasets, manager, ret)
   1149 elif isinstance(spec, DatasetSpec):

File ~/opt/miniconda3/lib/python3.9/site-packages/hdmf/build/objectmapper.py:1198, in ObjectMapper.__get_sub_builders(self, sub_builders, subspecs, manager, ret)
   1195     continue
   1196 if dt is None:
   1197     # recurse
-> 1198     ret.update(self.__get_subspec_values(sub_builder, subspec, manager))
   1199 else:
   1200     ret[subspec] = manager.construct(sub_builder)

File ~/opt/miniconda3/lib/python3.9/site-packages/hdmf/build/objectmapper.py:1147, in ObjectMapper.__get_subspec_values(self, builder, spec, manager)
   1145                 ret[subspec] = self.__flatten(sub_builder, subspec, manager)
   1146     # now process groups and datasets
-> 1147     self.__get_sub_builders(groups, spec.groups, manager, ret)
   1148     self.__get_sub_builders(datasets, spec.datasets, manager, ret)
   1149 elif isinstance(spec, DatasetSpec):

File ~/opt/miniconda3/lib/python3.9/site-packages/hdmf/build/objectmapper.py:1200, in ObjectMapper.__get_sub_builders(self, sub_builders, subspecs, manager, ret)
   1198     ret.update(self.__get_subspec_values(sub_builder, subspec, manager))
   1199 else:
-> 1200     ret[subspec] = manager.construct(sub_builder)

File ~/opt/miniconda3/lib/python3.9/site-packages/hdmf/utils.py:645, in docval.<locals>.dec.<locals>.func_call(*args, **kwargs)
    643 def func_call(*args, **kwargs):
    644     pargs = _check_args(args, kwargs)
--> 645     return func(args[0], **pargs)

File ~/opt/miniconda3/lib/python3.9/site-packages/hdmf/build/manager.py:276, in BuildManager.construct(self, **kwargs)
    274 if parent_builder is not None:
    275     parent = self._get_proxy_builder(parent_builder)
--> 276     result = self.__type_map.construct(builder, self, parent)
    277 else:
    278     # we are at the top of the hierarchy,
    279     # so it must be time to resolve parents
    280     result = self.__type_map.construct(builder, self, None)

File ~/opt/miniconda3/lib/python3.9/site-packages/hdmf/utils.py:645, in docval.<locals>.dec.<locals>.func_call(*args, **kwargs)
    643 def func_call(*args, **kwargs):
    644     pargs = _check_args(args, kwargs)
--> 645     return func(args[0], **pargs)

File ~/opt/miniconda3/lib/python3.9/site-packages/hdmf/build/manager.py:789, in TypeMap.construct(self, **kwargs)
    787     raise ValueError('No ObjectMapper found for builder of type %s' % dt)
    788 else:
--> 789     return obj_mapper.construct(builder, build_manager, parent)

File ~/opt/miniconda3/lib/python3.9/site-packages/hdmf/utils.py:645, in docval.<locals>.dec.<locals>.func_call(*args, **kwargs)
    643 def func_call(*args, **kwargs):
    644     pargs = _check_args(args, kwargs)
--> 645     return func(args[0], **pargs)

File ~/opt/miniconda3/lib/python3.9/site-packages/hdmf/build/objectmapper.py:1218, in ObjectMapper.construct(self, **kwargs)
   1216 cls = manager.get_cls(builder)
   1217 # gather all subspecs
-> 1218 subspecs = self.__get_subspec_values(builder, self.spec, manager)
   1219 # get the constructor argument that each specification corresponds to
   1220 const_args = dict()

File ~/opt/miniconda3/lib/python3.9/site-packages/hdmf/build/objectmapper.py:1147, in ObjectMapper.__get_subspec_values(self, builder, spec, manager)
   1145                 ret[subspec] = self.__flatten(sub_builder, subspec, manager)
   1146     # now process groups and datasets
-> 1147     self.__get_sub_builders(groups, spec.groups, manager, ret)
   1148     self.__get_sub_builders(datasets, spec.datasets, manager, ret)
   1149 elif isinstance(spec, DatasetSpec):

File ~/opt/miniconda3/lib/python3.9/site-packages/hdmf/build/objectmapper.py:1200, in ObjectMapper.__get_sub_builders(self, sub_builders, subspecs, manager, ret)
   1198     ret.update(self.__get_subspec_values(sub_builder, subspec, manager))
   1199 else:
-> 1200     ret[subspec] = manager.construct(sub_builder)

File ~/opt/miniconda3/lib/python3.9/site-packages/hdmf/utils.py:645, in docval.<locals>.dec.<locals>.func_call(*args, **kwargs)
    643 def func_call(*args, **kwargs):
    644     pargs = _check_args(args, kwargs)
--> 645     return func(args[0], **pargs)

File ~/opt/miniconda3/lib/python3.9/site-packages/hdmf/build/manager.py:276, in BuildManager.construct(self, **kwargs)
    274 if parent_builder is not None:
    275     parent = self._get_proxy_builder(parent_builder)
--> 276     result = self.__type_map.construct(builder, self, parent)
    277 else:
    278     # we are at the top of the hierarchy,
    279     # so it must be time to resolve parents
    280     result = self.__type_map.construct(builder, self, None)

File ~/opt/miniconda3/lib/python3.9/site-packages/hdmf/utils.py:645, in docval.<locals>.dec.<locals>.func_call(*args, **kwargs)
    643 def func_call(*args, **kwargs):
    644     pargs = _check_args(args, kwargs)
--> 645     return func(args[0], **pargs)

File ~/opt/miniconda3/lib/python3.9/site-packages/hdmf/build/manager.py:789, in TypeMap.construct(self, **kwargs)
    787     raise ValueError('No ObjectMapper found for builder of type %s' % dt)
    788 else:
--> 789     return obj_mapper.construct(builder, build_manager, parent)

File ~/opt/miniconda3/lib/python3.9/site-packages/hdmf/utils.py:645, in docval.<locals>.dec.<locals>.func_call(*args, **kwargs)
    643 def func_call(*args, **kwargs):
    644     pargs = _check_args(args, kwargs)
--> 645     return func(args[0], **pargs)

File ~/opt/miniconda3/lib/python3.9/site-packages/hdmf/build/objectmapper.py:1218, in ObjectMapper.construct(self, **kwargs)
   1216 cls = manager.get_cls(builder)
   1217 # gather all subspecs
-> 1218 subspecs = self.__get_subspec_values(builder, self.spec, manager)
   1219 # get the constructor argument that each specification corresponds to
   1220 const_args = dict()

File ~/opt/miniconda3/lib/python3.9/site-packages/hdmf/build/objectmapper.py:1147, in ObjectMapper.__get_subspec_values(self, builder, spec, manager)
   1145                 ret[subspec] = self.__flatten(sub_builder, subspec, manager)
   1146     # now process groups and datasets
-> 1147     self.__get_sub_builders(groups, spec.groups, manager, ret)
   1148     self.__get_sub_builders(datasets, spec.datasets, manager, ret)
   1149 elif isinstance(spec, DatasetSpec):

File ~/opt/miniconda3/lib/python3.9/site-packages/hdmf/build/objectmapper.py:1190, in ObjectMapper.__get_sub_builders(self, sub_builders, subspecs, manager, ret)
   1188     sub_builder = builder_dt.get(dt)
   1189     if sub_builder is not None:
-> 1190         sub_builder = self.__flatten(sub_builder, subspec, manager)
   1191         ret[subspec] = sub_builder
   1192 else:

File ~/opt/miniconda3/lib/python3.9/site-packages/hdmf/build/objectmapper.py:1203, in ObjectMapper.__flatten(self, sub_builder, subspec, manager)
   1202 def __flatten(self, sub_builder, subspec, manager):
-> 1203     tmp = [manager.construct(b) for b in sub_builder]
   1204     if len(tmp) == 1 and not subspec.is_many():
   1205         tmp = tmp[0]

File ~/opt/miniconda3/lib/python3.9/site-packages/hdmf/build/objectmapper.py:1203, in <listcomp>(.0)
   1202 def __flatten(self, sub_builder, subspec, manager):
-> 1203     tmp = [manager.construct(b) for b in sub_builder]
   1204     if len(tmp) == 1 and not subspec.is_many():
   1205         tmp = tmp[0]

File ~/opt/miniconda3/lib/python3.9/site-packages/hdmf/utils.py:645, in docval.<locals>.dec.<locals>.func_call(*args, **kwargs)
    643 def func_call(*args, **kwargs):
    644     pargs = _check_args(args, kwargs)
--> 645     return func(args[0], **pargs)

File ~/opt/miniconda3/lib/python3.9/site-packages/hdmf/build/manager.py:276, in BuildManager.construct(self, **kwargs)
    274 if parent_builder is not None:
    275     parent = self._get_proxy_builder(parent_builder)
--> 276     result = self.__type_map.construct(builder, self, parent)
    277 else:
    278     # we are at the top of the hierarchy,
    279     # so it must be time to resolve parents
    280     result = self.__type_map.construct(builder, self, None)

File ~/opt/miniconda3/lib/python3.9/site-packages/hdmf/utils.py:645, in docval.<locals>.dec.<locals>.func_call(*args, **kwargs)
    643 def func_call(*args, **kwargs):
    644     pargs = _check_args(args, kwargs)
--> 645     return func(args[0], **pargs)

File ~/opt/miniconda3/lib/python3.9/site-packages/hdmf/build/manager.py:789, in TypeMap.construct(self, **kwargs)
    787     raise ValueError('No ObjectMapper found for builder of type %s' % dt)
    788 else:
--> 789     return obj_mapper.construct(builder, build_manager, parent)

File ~/opt/miniconda3/lib/python3.9/site-packages/hdmf/utils.py:645, in docval.<locals>.dec.<locals>.func_call(*args, **kwargs)
    643 def func_call(*args, **kwargs):
    644     pargs = _check_args(args, kwargs)
--> 645     return func(args[0], **pargs)

File ~/opt/miniconda3/lib/python3.9/site-packages/hdmf/build/objectmapper.py:1252, in ObjectMapper.construct(self, **kwargs)
   1250 except Exception as ex:
   1251     msg = 'Could not construct %s object due to: %s' % (cls.__name__, ex)
-> 1252     raise ConstructError(builder, msg) from ex
   1253 return obj

ConstructError: (root/general/subject/cortical_surfaces/std_inflated_LH GroupBuilder {'attributes': {'namespace': 'ndx-ecog', 'neurodata_type': 'Surface', 'object_id': 'aac7d8fb-ad9e-4632-9fa3-f897392d846f'}, 'groups': {'parcellations': root/general/subject/cortical_surfaces/std_inflated_LH/parcellations GroupBuilder {'attributes': {'namespace': 'ndx-ecog', 'neurodata_type': 'Parcellations', 'object_id': '5475cfbc-cf80-4eaa-a616-e5b9560e7cd7'}, 'groups': {}, 'datasets': {'HCP': root/general/subject/cortical_surfaces/std_inflated_LH/parcellations/HCP DatasetBuilder {'attributes': {'labels': array(['???', 'V1', 'MST', 'V6', 'V2', 'V3', 'V4', 'V8', '4', '3b', 'FEF',
       'PEF', '55b', 'V3A', 'RSC', 'POS2', 'V7', 'IPS1', 'FFC', 'V3B',
       'LO1', 'LO2', 'PIT', 'MT', 'A1', 'PSL', 'SFL', 'PCV', 'STV', '7Pm',
       '7m', 'POS1', '23d', 'v23ab', 'd23ab', '31pv', '5m', '5mv', '23c',
       '5L', '24dd', '24dv', '7AL', 'SCEF', '6ma', '7Am', '7PL', '7PC',
       'LIPv', 'VIP', 'MIP', '1', '2', '3a', '6d', '6mp', '6v', 'p24pr',
       '33pr', 'a24pr', 'p32pr', 'a24', 'd32', '8BM', 'p32', '10r', '47m',
       '8Av', '8Ad', '9m', '8BL', '9p', '10d', '8C', '44', '45', '47l',
       'a47r', '6r', 'IFJa', 'IFJp', 'IFSp', 'IFSa', 'p9-46v', '46',
       'a9-46v', '9-46d', '9a', '10v', 'a10p', '10pp', '11l', '13l',
       'OFC', '47s', 'LIPd', '6a', 'i6-8', 's6-8', '43', 'OP4', 'OP1',
       'OP2-3', '52', 'RI', 'PFcm', 'PoI2', 'TA2', 'FOP4', 'MI', 'Pir',
       'AVI', 'AAIC', 'FOP1', 'FOP3', 'FOP2', 'PFt', 'AIP', 'EC', 'PreS',
       'H', 'ProS', 'PeEc', 'STGa', 'PBelt', 'A5', 'PHA1', 'PHA3',
       'STSda', 'STSdp', 'STSvp', 'TGd', 'TE1a', 'TE1p', 'TE2a', 'TF',
       'TE2p', 'PHT', 'PH', 'TPOJ1', 'TPOJ2', 'TPOJ3', 'DVT', 'PGp',
       'IP2', 'IP1', 'IP0', 'PFop', 'PF', 'PFm', 'PGi', 'PGs', 'V6A',
       'VMV1', 'VMV3', 'PHA2', 'V4t', 'FST', 'V3CD', 'LO3', 'VMV2',
       '31pd', '31a', 'VVC', '25', 's32', 'pOFC', 'PoI1', 'Ig', 'FOP5',
       'p10p', 'p47r', 'TGv', 'MBelt', 'LBelt', 'A4', 'STSva', 'TE1m',
       'PI', 'a32pr', 'p24'], dtype=object), 'namespace': 'ndx-ecog', 'neurodata_type': 'Parcellation', 'object_id': '10436e73-6d00-41e4-aa83-0490fcb78a3e'}, 'data': <Closed HDF5 dataset>}}, 'links': {}}}, 'datasets': {'faces': root/general/subject/cortical_surfaces/std_inflated_LH/faces DatasetBuilder {'attributes': {}, 'data': <Closed HDF5 dataset>}, 'vertices': root/general/subject/cortical_surfaces/std_inflated_LH/vertices DatasetBuilder {'attributes': {}, 'data': <Closed HDF5 dataset>}}, 'links': {}}, 'Could not construct Surface object due to: index of faces exceeds number vertices for std_inflated_LH. Faces should be 0-indexed, not 1-indexed')
bendichter commented 1 year ago

The file uses parcellations, which are defined in the extension.

I see a strange dataset in there though. What is "curv" doing here?

image
bendichter commented 1 year ago

As the traceback suggests, it looks like the "faces" array is 1-indexed instead of 0-indexed, since it goes from 1 to n_vertices+1. The solution is to

  1. Use and cache the latest version of ndx-ecog in the NWB file.
  2. subtract the faces array by 1 wherever it is currently 1-indexed.
Christonikos commented 1 year ago

Thanks, @bendichter—could you please explain a bit more the first step? I tried using the latest ndx-ecog version, but I still could not load the file. A code snippet with a requirements.txt would be very helpful.

bendichter commented 1 year ago

@Christonikos apologies for the sporadic support during the holiday break.

I was able to get past your error by pip-installing ndx-ecog and importing it. I'm not quite sure yet why your original attempt isn't working- it should. It may be revealing a weakness in our automatic class generation, but I am able to get past it for now with this import (import ndx_ecog) before the NWBHDF5IO. This produces another very long traceback but you'll notice at the end that the error is different. This traceback reveals that there is a problem with the data that is preventing this file from being opened.

We put a checker in to make sure that the faces array has the correct values. This array defines triangles between vertices by listing triplets of vertices for each triangle. The complex 3-D shapes are constructed from these triangles. These indices should be 0-indexed, meaning they should go from 0 to n_vertices - 1. This is a bit against the grain for MATLAB users but it is more common in the broader programming community and is the norm e.g. in Python. However the faces array here is clearly 1-indexed, since it has a range 1 to n_verticies.

I put a checker in the loader function to check for this condition and throw an error if it occurs. (Perhaps this is a bit heavy-handed. Maybe a warning would have been better than an error.) But the data is incorrect. All that would need to be done to correct these arrays is to subtract all of the faces values by 1. I could create a patch script but I think you are better off going to the data producers and requesting that they fix it on their side.

bendichter commented 1 year ago

I also don't know what's going on with the "curv" datasets

Christonikos commented 1 year ago

Hi @bendichter thanks for the follow-up and happy new year. I can confirm that I get the same traceback when importing ndx_ecog prior to NWBHDF5IO. What puzzles me is that I asked the hosting lab to load the data on their side, and they faced no such issue, nor did I when using an Ubuntu PC. I have also informed the hosting lab, so they can jump in the discussion and follow-up on their side.

bendichter commented 1 year ago

It's probably a package version issue. If they are using a version of ndx-ecog before that check was in place then they won't get that error. The data is still incorrect though even if they don't get the error, so I think the right way forward is to correct it.

Christonikos commented 1 year ago

@bendichter I am adding @OWoolnough in the conversation, as a representative of the hosting lab.

Christonikos commented 1 year ago

Hi everyone, I'm still unable to use these files on my M1, whereas in the past I was able to load and work with it (despite the faces index problem) and @owoolnough is able to load the dataset on his side. Any advice on how to proceed?

bendichter commented 1 year ago

@Christonikos and @owoolnough the data in the NWB file supplied is incorrect.

import h5py

file = h5py.File("/Users/bendichter/Downloads/sub-TS106_places_ieeg_2.nwb")
print(np.min(file["/general/subject/cortical_surfaces/std_pial_LH/faces"]))
print(np.max(file["/general/subject/cortical_surfaces/std_pial_LH/faces"]))
print(file["/general/subject/cortical_surfaces/std_pial_LH/vertices"].shape)
1
156252
(156252, 3)

faces should take values from 0 to shape[0] - 1. That needs to be fixed before this is readable. It is possible that you were able to read this file using an outdated ndx-ecog package on another computer, but the data is still wrong and I don't think it has to do with M1.