unit type for InternalOnePhoton necessary and erroneous flagging of data orientation

[marl1bakker]

Hi

I'm trying to upload my data to DANDI, and ran into some problems that I think would give other users issues as well.

The first problem was that even though data_unit is not a required field in the InternalOnePhoton file (https://neurodatawithoutborders.github.io/matnwb/doc/+types/+core/OnePhotonSeries.html), the data inspector cannot read the file when the output of the units is empty. Running the inspector in linux gave me the following error:

nwbinspector /home/mbakker/P2_scripts/MarleenP2/ExampleData/M32-A1-R2/dat-nwb/ --config dandi --report-file-path /home/mbakker/P2_scripts/xxx.txt Traceback (most recent call last): File "/home/mbakker/anaconda3/envs/nwb/lib/python3.10/site-packages/hdmf/build/objectmapper.py", line 1360, in construct obj = self.new_container(cls, builder.source, parent, builder.attributes.get(self.spec.id_key()), File "/home/mbakker/anaconda3/envs/nwb/lib/python3.10/site-packages/hdmf/build/objectmapper.py", line 1373, in new_container obj.init__(**kwargs) File "/home/mbakker/anaconda3/envs/nwb/lib/python3.10/site-packages/hdmf/utils.py", line 667, in func_call pargs = _check_args(args, kwargs) File "/home/mbakker/anaconda3/envs/nwb/lib/python3.10/site-packages/hdmf/utils.py", line 660, in _check_args raise ExceptionType(msg) TypeError: OnePhotonSeries.init: incorrect type for 'unit' (got 'ndarray', expected 'str')

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/mbakker/anaconda3/envs/nwb/bin/nwbinspector", line 10, in sys.exit(_inspect_all_cli()) File "/home/mbakker/anaconda3/envs/nwb/lib/python3.10/site-packages/click/core.py", line 1157, in call return self.main(args, kwargs) File "/home/mbakker/anaconda3/envs/nwb/lib/python3.10/site-packages/click/core.py", line 1078, in main rv = self.invoke(ctx) File "/home/mbakker/anaconda3/envs/nwb/lib/python3.10/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, ctx.params) File "/home/mbakker/anaconda3/envs/nwb/lib/python3.10/site-packages/click/core.py", line 783, in invoke return __callback(args, *kwargs) File "/home/mbakker/anaconda3/envs/nwb/lib/python3.10/site-packages/nwbinspector/_inspection_cli.py", line 121, in _inspect_all_cli messages = list( File "/home/mbakker/anaconda3/envs/nwb/lib/python3.10/site-packages/nwbinspector/_inspection.py", line 140, in inspect_all nwbfile = robust_s3_read(io.read) File "/home/mbakker/anaconda3/envs/nwb/lib/python3.10/site-packages/nwbinspector/utils/_utils.py", line 181, in robust_s3_read raise exc File "/home/mbakker/anaconda3/envs/nwb/lib/python3.10/site-packages/nwbinspector/utils/_utils.py", line 176, in robust_s3_read return command(command_args, command_kwargs) File "/home/mbakker/anaconda3/envs/nwb/lib/python3.10/site-packages/hdmf/utils.py", line 668, in func_call return func(args[0], pargs) File "/home/mbakker/anaconda3/envs/nwb/lib/python3.10/site-packages/pynwb/init.py", line 361, in read file = super().read(kwargs) File "/home/mbakker/anaconda3/envs/nwb/lib/python3.10/site-packages/hdmf/backends/hdf5/h5tools.py", line 500, in read return super().read(kwargs) File "/home/mbakker/anaconda3/envs/nwb/lib/python3.10/site-packages/hdmf/utils.py", line 668, in func_call return func(args[0], pargs) File "/home/mbakker/anaconda3/envs/nwb/lib/python3.10/site-packages/hdmf/backends/io.py", line 60, in read container = self.__manager.construct(f_builder) File "/home/mbakker/anaconda3/envs/nwb/lib/python3.10/site-packages/hdmf/utils.py", line 668, in func_call return func(args[0], pargs) File "/home/mbakker/anaconda3/envs/nwb/lib/python3.10/site-packages/hdmf/build/manager.py", line 286, in construct result = self.type_map.construct(builder, self, None) File "/home/mbakker/anaconda3/envs/nwb/lib/python3.10/site-packages/hdmf/utils.py", line 668, in func_call return func(args[0], pargs) File "/home/mbakker/anaconda3/envs/nwb/lib/python3.10/site-packages/hdmf/build/manager.py", line 827, in construct return obj_mapper.construct(builder, build_manager, parent) File "/home/mbakker/anaconda3/envs/nwb/lib/python3.10/site-packages/hdmf/utils.py", line 668, in func_call return func(args[0], pargs) File "/home/mbakker/anaconda3/envs/nwb/lib/python3.10/site-packages/hdmf/build/objectmapper.py", line 1330, in construct subspecs = self.get_subspec_values(builder, self.spec, manager) File "/home/mbakker/anaconda3/envs/nwb/lib/python3.10/site-packages/hdmf/build/objectmapper.py", line 1259, in get_subspec_values self.get_sub_builders(groups, spec.groups, manager, ret) File "/home/mbakker/anaconda3/envs/nwb/lib/python3.10/site-packages/hdmf/build/objectmapper.py", line 1310, in get_sub_builders ret.update(self.__get_subspec_values(sub_builder, subspec, manager)) File "/home/mbakker/anaconda3/envs/nwb/lib/python3.10/site-packages/hdmf/build/objectmapper.py", line 1259, in get_subspec_values self.get_sub_builders(groups, spec.groups, manager, ret) File "/home/mbakker/anaconda3/envs/nwb/lib/python3.10/site-packages/hdmf/build/objectmapper.py", line 1302, in get_sub_builders sub_builder = self.flatten(sub_builder, subspec, manager) File "/home/mbakker/anaconda3/envs/nwb/lib/python3.10/site-packages/hdmf/build/objectmapper.py", line 1315, in __flatten tmp = [manager.construct(b) for b in sub_builder] File "/home/mbakker/anaconda3/envs/nwb/lib/python3.10/site-packages/hdmf/build/objectmapper.py", line 1315, in tmp = [manager.construct(b) for b in sub_builder] File "/home/mbakker/anaconda3/envs/nwb/lib/python3.10/site-packages/hdmf/utils.py", line 668, in func_call return func(args[0], **pargs) File "/home/mbakker/anaconda3/envs/nwb/lib/python3.10/site-packages/hdmf/build/manager.py", line 282, in construct result = self.type_map.construct(builder, self, parent) File "/home/mbakker/anaconda3/envs/nwb/lib/python3.10/site-packages/hdmf/utils.py", line 668, in func_call return func(args[0], pargs) File "/home/mbakker/anaconda3/envs/nwb/lib/python3.10/site-packages/hdmf/build/manager.py", line 827, in construct return obj_mapper.construct(builder, build_manager, parent) File "/home/mbakker/anaconda3/envs/nwb/lib/python3.10/site-packages/hdmf/utils.py", line 668, in func_call return func(args[0], pargs) File "/home/mbakker/anaconda3/envs/nwb/lib/python3.10/site-packages/hdmf/build/objectmapper.py", line 1364, in construct raise ConstructError(builder, msg) from ex hdmf.build.errors.ConstructError: (root/acquisition/1pInternal GroupBuilder {'attributes': {'binning': 2, 'comments': 'no comments', 'description': 'no description', 'exposure_time': 1.0, 'namespace': 'core', 'neurodata_type': 'OnePhotonSeries', 'object_id': 'ccc27f0d-6bc3-4345-96ee-c62c093c088d'}, 'groups': {}, 'datasets': {'data': root/acquisition/1pInternal/data DatasetBuilder {'attributes': {'conversion': 1.0, 'offset': 0.0, 'resolution': -1.0, 'unit': array([], dtype=object)}, 'data': }, 'starting_time': root/acquisition/1pInternal/starting_time DatasetBuilder {'attributes': {'rate': 15.0, 'unit': 'seconds'}, 'data': 0.0}}, 'links': {'imaging_plane': root/acquisition/1pInternal/imaging_plane LinkBuilder {'builder': root/general/optophysiology/imaging_plane GroupBuilder {'attributes': {'namespace': 'core', 'neurodata_type': 'ImagingPlane', 'object_id': '6dc31388-5c7b-42f0-bddb-4fa82a7da222'}, 'groups': {'optical_channel': root/general/optophysiology/imaging_plane/optical_channel GroupBuilder {'attributes': {'namespace': 'core', 'neurodata_type': 'OpticalChannel', 'object_id': 'e03696ae-53c5-422b-98a0-cda5727bf73c'}, 'groups': {}, 'datasets': {'description': root/general/optophysiology/imaging_plane/optical_channel/description DatasetBuilder {'attributes': {}, 'data': 'Red'}, 'emission_lambda': root/general/optophysiology/imaging_plane/optical_channel/emission_lambda DatasetBuilder {'attributes': {}, 'data': 620.0}}, 'links': {}}}, 'datasets': {'description': root/general/optophysiology/imaging_plane/description DatasetBuilder {'attributes': {}, 'data': 'whole brain'}, 'excitation_lambda': root/general/optophysiology/imaging_plane/excitation_lambda DatasetBuilder {'attributes': {}, 'data': 620.0}, 'imaging_rate': root/general/optophysiology/imaging_plane/imaging_rate DatasetBuilder {'attributes': {}, 'data': 15.0}, 'indicator': root/general/optophysiology/imaging_plane/indicator DatasetBuilder {'attributes': {}, 'data': 'Red'}, 'location': root/general/optophysiology/imaging_plane/location DatasetBuilder {'attributes': {}, 'data': 'cortex'}}, 'links': {'device': root/general/optophysiology/imaging_plane/device LinkBuilder {'builder': root/general/devices/Device GroupBuilder {'attributes': {'description': 'Camera Model: CS2100M on LightTrackOiS200 system', 'manufacturer': 'Labeo Technologies', 'namespace': 'core', 'neurodata_type': 'Device', 'object_id': 'd15271b4-6f22-46cf-a4c1-8121b4ad84c6'}, 'groups': {}, 'datasets': {}, 'links': {}}}}}}}}, "Could not construct OnePhotonSeries object due to: OnePhotonSeries.init: incorrect type for 'unit' (got 'ndarray', expected 'str')")

This error disappeared when I specified data_unit as '-'. It might be useful to either set that as a required field or to make sure that it's not read as an empty array.

Secondly, I receive a "best practice violation" message when inspecting my data based on the orientation of my data. I export my data in matlab with the time axis as first axis, and it is the longest axis as well. The data in matlab before exporting to nwb has the size 9007x512x512. Similarly, when loading the nwb file in matlab, it loads the data as 9007x512x512. However, I get the error message that the longest axis is not the first: 

1 BEST_PRACTICE_VIOLATION

=========================

1.2 /home/mbakker/P2_scripts/MarleenP2/ExampleData/M32-A1-R2/dat-nwb/fluo_567.nwb and 2 other files: check_data_orientation - 'OnePhotonSeries' object at location '/acquisition/1pInternal' Message: Data may be in the wrong orientation. Time should be in the first dimension, and is usually the longest dimension. Here, another dimension is longer.

How can this be? Is my data correctly exported?

Best wishes, Marleen

[kabilar]

Hi @bendichter, could you please look into this issue? Thank you.