search

hdmf-dev / hdmf

The Hierarchical Data Modeling Framework
http://hdmf.readthedocs.io
Other
47 stars 26 forks source link

[Bug]: Regression in fresh 3.14.4 #1186

Closed yarikoptic closed 1 month ago

yarikoptic commented 1 month ago

What happened?

More of background in https://github.com/dandi/dandi-cli/issues/1494#issuecomment-2332453413 and regression was introduced in https://github.com/hdmf-dev/hdmf/commit/2b167aedc8a8f58afd75d3d0c750f6d620dc663d

"copied" using

    with pynwb.NWBHDF5IO(src, "r") as ior, pynwb.NWBHDF5IO(dest, "w") as iow:
        data = ior.read()
        data.generate_new_id()
        iow.export(ior, nwbfile=data)

gains some "unique" and odd thing expressed by h5py as

❯ grep StrDataset /home/yoh/.tmp/pytest-of-yoh/pytest-785/simple20/simple2.dump /home/yoh/.tmp/pytest-of-yoh/pytest-785/test_ambiguous0/simple2.dump
/home/yoh/.tmp/pytest-of-yoh/pytest-785/test_ambiguous0/simple2.dump:         (0): "<StrDataset for HDF5 dataset "keywords": shape (2,), type "|O">"

Steps to Reproduce

and here is a full reproducer script

#!/usr/bin/env python3

import pynwb
import sys

src, dest = sys.argv[1:3]
print(f"Copying {src} {dest}")

with pynwb.NWBHDF5IO(src, "r") as ior, pynwb.NWBHDF5IO(dest, "w") as iow:
    data = ior.read()
    data.generate_new_id()
    iow.export(ior, nwbfile=data)

print(f"Now reading {dest}")

with pynwb.NWBHDF5IO(dest, "r") as iow:
    data = iow.read()

which if ran on this file

http://www.oneukrainian.com/tmp/simple2.nwb

would result in

❯ /tmp/simple2.py /tmp/simple2.nwb /tmp/simple2-copy.nwb
Copying /tmp/simple2.nwb /tmp/simple2-copy.nwb
Now reading /tmp/simple2-copy.nwb
Traceback (most recent call last):
  File "/home/yoh/deb/gits/pkg-exppsy/hdmf-upstream/src/hdmf/build/objectmapper.py", line 1357, in construct
    obj = self.__new_container__(cls, builder.source, parent, builder.attributes.get(self.__spec.id_key()),
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yoh/deb/gits/pkg-exppsy/hdmf-upstream/src/hdmf/build/objectmapper.py", line 1370, in __new_container__
    obj.__init__(**kwargs)
  File "/home/yoh/deb/gits/pkg-exppsy/hdmf-upstream/src/hdmf/utils.py", line 667, in func_call
    pargs = _check_args(args, kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yoh/deb/gits/pkg-exppsy/hdmf-upstream/src/hdmf/utils.py", line 660, in _check_args
    raise ExceptionType(msg)
TypeError: NWBFile.__init__: incorrect type for 'keywords' (got 'str', expected 'ndarray, list, tuple, Dataset, Array, StrDataset, HDMFDataset or AbstractDataChunkIterator')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/tmp/simple2.py", line 17, in <module>
    data = iow.read()
           ^^^^^^^^^^
  File "/home/yoh/deb/gits/pkg-exppsy/hdmf-upstream/src/hdmf/utils.py", line 668, in func_call
    return func(args[0], **pargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yoh/proj/dandi/dandi-cli-master/venvs/dev3.12/lib/python3.12/site-packages/pynwb/__init__.py", line 326, in read
    file = super().read(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yoh/deb/gits/pkg-exppsy/hdmf-upstream/src/hdmf/backends/hdf5/h5tools.py", line 500, in read
    return super().read(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yoh/deb/gits/pkg-exppsy/hdmf-upstream/src/hdmf/utils.py", line 668, in func_call
    return func(args[0], **pargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yoh/deb/gits/pkg-exppsy/hdmf-upstream/src/hdmf/backends/io.py", line 60, in read
    container = self.__manager.construct(f_builder)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yoh/deb/gits/pkg-exppsy/hdmf-upstream/src/hdmf/utils.py", line 668, in func_call
    return func(args[0], **pargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yoh/deb/gits/pkg-exppsy/hdmf-upstream/src/hdmf/build/manager.py", line 286, in construct
    result = self.__type_map.construct(builder, self, None)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yoh/deb/gits/pkg-exppsy/hdmf-upstream/src/hdmf/utils.py", line 668, in func_call
    return func(args[0], **pargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yoh/deb/gits/pkg-exppsy/hdmf-upstream/src/hdmf/build/manager.py", line 827, in construct
    return obj_mapper.construct(builder, build_manager, parent)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yoh/deb/gits/pkg-exppsy/hdmf-upstream/src/hdmf/utils.py", line 668, in func_call
    return func(args[0], **pargs)
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/yoh/deb/gits/pkg-exppsy/hdmf-upstream/src/hdmf/build/objectmapper.py", line 1361, in construct
    raise ConstructError(builder, msg) from ex
hdmf.build.errors.ConstructError: (root GroupBuilder {'attributes': {'namespace': 'core', 'neurodata_type': 'NWBFile', 'nwb_version': '2.7.0', 'object_id': '85f9aa7e-36fc-449d-8abb-903e7826ead6'}, 'groups': {'acquisition': root/acquisition GroupBuilder {'attributes': {}, 'groups': {}, 'datasets': {}, 'links': {}}, 'analysis': root/analysis GroupBuilder {'attributes': {}, 'groups': {}, 'datasets': {}, 'links': {}}, 'general': root/general GroupBuilder {'attributes': {}, 'groups': {'subject': root/general/subject GroupBuilder {'attributes': {'namespace': 'core', 'neurodata_type': 'Subject', 'object_id': '6e074def-32d9-4fcd-97b8-4c851598cde6'}, 'groups': {}, 'datasets': {'date_of_birth': root/general/subject/date_of_birth DatasetBuilder {'attributes': {}, 'data': '2016-12-01T00:00:00+00:00'}, 'sex': root/general/subject/sex DatasetBuilder {'attributes': {}, 'data': 'U'}, 'species': root/general/subject/species DatasetBuilder {'attributes': {}, 'data': 'Mus musculus'}, 'subject_id': root/general/subject/subject_id DatasetBuilder {'attributes': {}, 'data': 'mouse001'}}, 'links': {}}}, 'datasets': {'experiment_description': root/general/experiment_description DatasetBuilder {'attributes': {}, 'data': 'experiment_description1'}, 'experimenter': root/general/experimenter DatasetBuilder {'attributes': {}, 'data': <StrDataset for Closed HDF5 dataset>}, 'institution': root/general/institution DatasetBuilder {'attributes': {}, 'data': 'institution1'}, 'keywords': root/general/keywords DatasetBuilder {'attributes': {}, 'data': '<StrDataset for HDF5 dataset "keywords": shape (2,), type "|O">'}, 'lab': root/general/lab DatasetBuilder {'attributes': {}, 'data': 'lab1'}, 'related_publications': root/general/related_publications DatasetBuilder {'attributes': {}, 'data': <StrDataset for Closed HDF5 dataset>}, 'session_id': root/general/session_id DatasetBuilder {'attributes': {}, 'data': 'session_id1'}}, 'links': {}}, 'processing': root/processing GroupBuilder {'attributes': {}, 'groups': {}, 'datasets': {}, 'links': {}}, 'stimulus': root/stimulus GroupBuilder {'attributes': {}, 'groups': {'presentation': root/stimulus/presentation GroupBuilder {'attributes': {}, 'groups': {}, 'datasets': {}, 'links': {}}, 'templates': root/stimulus/templates GroupBuilder {'attributes': {}, 'groups': {}, 'datasets': {}, 'links': {}}}, 'datasets': {}, 'links': {}}}, 'datasets': {'file_create_date': root/file_create_date DatasetBuilder {'attributes': {}, 'data': <Closed HDF5 dataset>}, 'identifier': root/identifier DatasetBuilder {'attributes': {}, 'data': '08f490263d06428985b8529d0a016908'}, 'session_description': root/session_description DatasetBuilder {'attributes': {}, 'data': 'session_description1'}, 'session_start_time': root/session_start_time DatasetBuilder {'attributes': {}, 'data': '2017-04-15T12:00:00+00:00'}, 'timestamps_reference_time': root/timestamps_reference_time DatasetBuilder {'attributes': {}, 'data': '2017-04-15T12:00:00+00:00'}}, 'links': {}}, "Could not construct NWBFile object due to: NWBFile.__init__: incorrect type for 'keywords' (got 'str', expected 'ndarray, list, tuple, Dataset, Array, StrDataset, HDMFDataset or AbstractDataChunkIterator')")
/tmp/simple2.py /tmp/simple2.nwb /tmp/simple2-copy.nwb  3.87s user 1.33s system 281% cpu 1.844 total

Traceback

No response

Operating System

Linux

Python Executable

Python

Python Version

3.12

Package Versions

No response

rly commented 1 month ago

Thanks for the bug report @yarikoptic and the great MWE. We created a fix in https://github.com/hdmf-dev/hdmf/pull/1189.