[Bug]: Can't Export NWB

rcpeene commented 1 month ago

What happened?

I am appending columns to the NWB electrodes table and then exporting it as a new file, described in more detail here https://github.com/hdmf-dev/hdmf-zarr/issues/179

I receive the error below when trying to export the NWB. Is this an hdmf problem? The same code works when using hdmf-zarr to export.

Steps to Reproduce

Use pynwb and run this code where locs, xs, ys, and zs are numpy arrays

    with io_class(str(scratch_nwb_path), mode='a') as read_io:
        nwb = read_io.read()

        print('Getting new electrode columns')
        locs, xs, ys, zs = get_new_electrode_colums(nwb, ccf_map)

        nwb.electrodes.location.data[:] = np.array(locs)
        nwb.electrodes.add_column('x', 'ccf x coordinate', data=xs)
        nwb.electrodes.add_column('y', 'ccf y coordinate', data=ys)
        nwb.electrodes.add_column('z', 'ccf z coordinate', data=zs)

        print('Exporting to NWB:',result_nwb_path)
        with io_class(str(result_nwb_path), "w") as export_io:
            export_io.export(src_io=read_io, nwbfile=nwb, write_args={'link_data': False})
        print(f"Done writing {result_nwb_path}")

Traceback

Traceback (most recent call last):
  File "/code/run_capsule.py", line 126, in <module>
    if __name__ == "__main__": run()
  File "/code/run_capsule.py", line 122, in run
    export_io.export(src_io=read_io, nwbfile=nwb, write_args={'link_data': False})
  File "/opt/conda/lib/python3.9/site-packages/hdmf/utils.py", line 668, in func_call
    return func(args[0], **pargs)
  File "/opt/conda/lib/python3.9/site-packages/pynwb/__init__.py", line 410, in export
    super().export(**kwargs)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/utils.py", line 668, in func_call
    return func(args[0], **pargs)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/backends/hdf5/h5tools.py", line 458, in export
    super().export(**ckwargs)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/utils.py", line 668, in func_call
    return func(args[0], **pargs)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/backends/io.py", line 163, in export
    bldr = src_io.manager.build(container, source=self.__source, root=True, export=True)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/utils.py", line 668, in func_call
    return func(args[0], **pargs)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/build/manager.py", line 173, in build
    result = self.__type_map.build(container, self, source=source, spec_ext=spec_ext, export=export)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/utils.py", line 668, in func_call
    return func(args[0], **pargs)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/build/manager.py", line 802, in build
    builder = obj_mapper.build(container, manager, builder=builder, source=source, spec_ext=spec_ext, export=export)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/utils.py", line 668, in func_call
    return func(args[0], **pargs)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/build/objectmapper.py", line 730, in build
    self.__add_groups(builder, self.__spec.groups, container, manager, source, export)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/build/objectmapper.py", line 1139, in __add_groups
    self.__add_groups(sub_builder, spec.groups, container, build_manager, source, export)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/build/objectmapper.py", line 1152, in __add_groups
    self.__add_containers(builder, spec, attr_value, build_manager, source, container, export)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/build/objectmapper.py", line 1210, in __add_containers
    self.__add_containers(builder, spec, container, build_manager, source, parent_container, export)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/build/objectmapper.py", line 1172, in __add_containers
    new_builder = build_manager.build(value, source=source, spec_ext=spec, export=export)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/utils.py", line 668, in func_call
    return func(args[0], **pargs)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/build/manager.py", line 173, in build
    result = self.__type_map.build(container, self, source=source, spec_ext=spec_ext, export=export)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/utils.py", line 668, in func_call
    return func(args[0], **pargs)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/build/manager.py", line 802, in build
    builder = obj_mapper.build(container, manager, builder=builder, source=source, spec_ext=spec_ext, export=export)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/utils.py", line 668, in func_call
    return func(args[0], **pargs)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/build/objectmapper.py", line 729, in build
    self.__add_datasets(builder, self.__spec.datasets, container, manager, source, export)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/build/objectmapper.py", line 1122, in __add_datasets
    self.__add_containers(builder, spec, attr_value, build_manager, source, container, export)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/build/objectmapper.py", line 1172, in __add_containers
    new_builder = build_manager.build(value, source=source, spec_ext=spec, export=export)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/utils.py", line 668, in func_call
    return func(args[0], **pargs)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/build/manager.py", line 173, in build
    result = self.__type_map.build(container, self, source=source, spec_ext=spec_ext, export=export)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/utils.py", line 668, in func_call
    return func(args[0], **pargs)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/build/manager.py", line 802, in build
    builder = obj_mapper.build(container, manager, builder=builder, source=source, spec_ext=spec_ext, export=export)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/utils.py", line 668, in func_call
    return func(args[0], **pargs)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/build/objectmapper.py", line 798, in build
    builder = DatasetBuilder(
  File "/opt/conda/lib/python3.9/site-packages/hdmf/utils.py", line 667, in func_call
    pargs = _check_args(args, kwargs)
  File "/opt/conda/lib/python3.9/site-packages/hdmf/utils.py", line 660, in _check_args
    raise ExceptionType(msg)
TypeError: DatasetBuilder.__init__: incorrect type for 'data' (got 'AstypeWrapper', expected 'ndarray, list, tuple, Dataset, Array, StrDataset, HDMFDataset, AbstractDataChunkIterator, str, int, float, bytes, bool, DataIO, DatasetBuilder, RegionBuilder, Iterable, datetime or date')

Operating System

Windows

Python Executable

Conda

Python Version

3.9

Package Versions

pynwb==2.8.2

Code of Conduct

[X] I agree to follow this project's Code of Conduct
[X] Have you checked the Contributing document?
[X] Have you ensured this bug was not already reported?

stephprince commented 1 month ago

Hi @rcpeene, I believe this would be an hdmf issue, but I wasn't able to reproduce the error with a small test example so I’m not positive what’s causing the error.

@mavaylon1 would you be able to take a look since you have been working on export?

mavaylon1 commented 1 month ago

@stephprince happy to help @rcpeene I will try the test code you gave to see if I can reproduce the error.

mavaylon1 commented 4 weeks ago

@rcpeene Are you exporting across backends or within the same? It looks like you are using the same io_class throughout

rcpeene commented 4 weeks ago

The same backends; I am opening in h5py and exporting to h5py in the run that is failing. When I change io_class to zarr_hdmf, then the run passes and the NWB exports.

rcpeene commented 3 weeks ago

Any more updates here? Currently holding back our pipeline

rly commented 3 weeks ago

@rcpeene, could you share the file that is causing this error? We cannot seem to reproduce the error on our end.

Secondly, are the xs, ys, and zs arrays 1-D float numpy arrays?

Can you also tell us what version of pynwb and hdmf you are running? Thank you

rcpeene commented 3 weeks ago

The file is an asset on code ocean. I could ask for permission to download it and share it. Would emailing it be an option? It will be rather large.
That is correct, they are 1D numpy arrays of floats.
Using hdmf-zarr 0.8.0 and pynwb 2.8.2

stephprince commented 2 weeks ago

If you could get permission to download and share that would be great! Email is an option or a google drive link would work as well.

rcpeene commented 2 weeks ago

@stephprince, @rly. I have shared with you the folder containing the nwb and a CCF file, and a link to the capsule repo :)

stephprince commented 2 weeks ago

@rcpeene thanks for sharing the file! I was able to reproduce your error. This issue was caused by a bug in HDMF during export of a specific case of a string dataset.

I am planning to get the related fix into the next hdmf release, which should occur sometime next week.

rcpeene commented 1 week ago

Great! Thanks for looking into it

NeurodataWithoutBorders / pynwb