NeurodataWithoutBorders / matnwb

A Matlab interface for reading and writing NWB files
BSD 2-Clause "Simplified" License
49 stars 32 forks source link

[Bug]: loss of relevant time zone information in datetime objects #558

Closed NeuroGuth closed 5 months ago

NeuroGuth commented 5 months ago

What happened?

Hello,

I have an issue reading in .nwb files in Python which I have created in Matlab.

I tried to assign datetime objects to the properties 'timestamps_reference_time' and 'session_start_time' but it would always result in the assignment of cells containing the datetime objects.

When I export an .nwb-file in Matlab, I cannot read it in using Python. I get the following error message:

"Could not construct NWBFile object due to: 'timestamps_reference_time' must be a timezone-aware datetime object."

I think that there might be an issue in the assignment of the datetime object to the nwbFile.

Any help will be greatly apprechiated!

Steps to Reproduce

% download an example nwb-File, e.g.: https://dandiarchive.org/dandiset/000469/0.240123.1806

% load file
File            = nwbRead('sub-1_ses-1_ecephys+image.nwb');

% display File.timestamps_reference_time
>> File.timestamps_reference_time
ans = 
  datetime
   2016-01-01

% reassign datetime objects
File.timestamps_reference_time = File.timestamps_reference_time;
File.session_start_time               = File.session_start_time;

% display File.timestamps_reference_time
>> File.timestamps_reference_time
ans =
  1×1 cell array
    {[2016-01-01]}

% export nwb-File
nwbExport(File, 'test.nwb');

Error Message

ConstructError: (root GroupBuilder {'attributes': {'namespace': 'core', 'neurodata_type': 'NWBFile', 'nwb_version': '2.6.0', 'object_id': '67b0c391-9d20-491b-a5b3-e6184302636b'}, 'groups': {'acquisition': root/acquisition GroupBuilder {'attributes': {}, 'groups': {'events': root/acquisition/events GroupBuilder {'attributes': {'comments': 'no comments', 'description': 'The events coorespond to the TTL markers for each trial. The TTL markers are the following: 61 = Start of Experiment, 11 = Fixation Cross, 1 = Picture #1 Shown, 2 = Picture #2 Shown, 3 = Picture #3 Shown, 5 = Transition between each picture presentation, 6 = End of Encoding Sequence / Start of Maintenance Period, 7 = Probe Stimulus, 8 = Subject Response, 60 = End of Experiment ', 'namespace': 'core', 'neurodata_type': 'TimeSeries', 'object_id': '8284e4a1-a7cf-4931-bc92-776022f442d0'}, 'groups': {}, 'datasets': {'data': root/acquisition/events/data DatasetBuilder {'attributes': {'conversion': 1.0, 'offset': 0.0, 'resolution': -1.0, 'unit': 'NA'}, 'data': <Closed HDF5 dataset>}, 'timestamps': root/acquisition/events/timestamps DatasetBuilder {'attributes': {'interval': 1, 'unit': 'seconds'}, 'data': <Closed HDF5 dataset>}}, 'links': {}}}, 'datasets': {}, 'links': {}}, 'analysis': root/analysis GroupBuilder {'attributes': {}, 'groups': {}, 'datasets': {}, 'links': {}}, 'general': root/general GroupBuilder {'attributes': {}, 'groups': {'devices': root/general/devices GroupBuilder {'attributes': {}, 'groups': {'NLX-microwires-41': root/general/devices/NLX-microwires-41 GroupBuilder {'attributes': {'description': 'Recordings were performed with Macro-Micro Hybrid Depth Electrodes with Behnke Fried/Micro Inner Wire Bundle in which each individual microwire has a diameter of 40 microns. Likwise, each Depth Electrode has 8 microwires.', 'namespace': 'core', 'neurodata_type': 'Device', 'object_id': '2b59e6b7-2a1c-431e-a505-373a1092ede1'}, 'groups': {}, 'datasets': {}, 'links': {}}, 'NLX-microwires-47': root/general/devices/NLX-microwires-47 GroupBuilder {'attributes': {'description': 'Recordings were performed with Macro-Micro Hybrid Depth Electrodes with Behnke Fried/Micro Inner Wire Bundle in which each individual microwire has a diameter of 40 microns. Likwise, each Depth Electrode has 8 microwires.', 'namespace': 'core', 'neurodata_type': 'Device', 'object_id': '7346546c-0729-42b9-9519-cf2a542eb805'}, 'groups': {}, 'datasets': {}, 'links': {}}, 'NLX-microwires-48': root/general/devices/NLX-microwires-48 GroupBuilder {'attributes': {'description': 'Recordings were performed with Macro-Micro Hybrid Depth Electrodes with Behnke Fried/Micro Inner Wire Bundle in which each individual microwire has a diameter of 40 microns. Likwise, each Depth Electrode has 8 microwires.', 'namespace': 'core', 'neurodata_type': 'Device', 'object_id': '218d840d-123f-41dd-83f3-61fa363de4b2'}, 'groups': {}, 'datasets': {}, 'links': {}}, 'NLX-microwires-54': root/general/devices/NLX-microwires-54 GroupBuilder {'attributes': {'description': 'Recordings were performed with Macro-Micro Hybrid Depth Electrodes with Behnke Fried/Micro Inner Wire Bundle in which each individual microwire has a diameter of 40 microns. Likwise, each Depth Electrode has 8 microwires.', 'namespace': 'core', 'neurodata_type': 'Device', 'object_id': '417ba654-5f7e-4a98-b0fe-653c41b702d7'}, 'groups': {}, 'datasets': {}, 'links': {}}, 'NLX-microwires-7': root/general/devices/NLX-microwires-7 GroupBuilder {'attributes': {'description': 'Recordings were performed with Macro-Micro Hybrid Depth Electrodes with Behnke Fried/Micro Inner Wire Bundle in which each individual microwire has a diameter of 40 microns. Likwise, each Depth Electrode has 8 microwires.', 'namespace': 'core', 'neurodata_type': 'Device', 'object_id': '057a24c3-68d3-4af0-a8a2-7c613ec28b1b'}, 'groups': {}, 'datasets': {}, 'links': {}}}, 'datasets': {}, 'links': {}}, 'extracellular_ephys': root/general/extracellular_ephys GroupBuilder {'attributes': {}, 'groups': {'NLX-microwires-41': root/general/extracellular_ephys/NLX-microwires-41 GroupBuilder {'attributes': {'description': 'Microwire', 'location': 'pre_supplementary_motor_area_right', 'namespace': 'core', 'neurodata_type': 'ElectrodeGroup', 'object_id': '44586e61-c750-4357-ba4f-3f5c0deb814b'}, 'groups': {}, 'datasets': {}, 'links': {'device': root/general/extracellular_ephys/NLX-microwires-41/device LinkBuilder {'builder': root/general/devices/NLX-microwires-41 GroupBuilder {'attributes': {'description': 'Recordings were performed with Macro-Micro Hybrid Depth Electrodes with Behnke Fried/Micro Inner Wire Bundle in which each individual microwire has a diameter of 40 microns. Likwise, each Depth Electrode has 8 microwires.', 'namespace': 'core', 'neurodata_type': 'Device', 'object_id': '2b59e6b7-2a1c-431e-a505-373a1092ede1'}, 'groups': {}, 'datasets': {}, 'links': {}}}}}, 'NLX-microwires-47': root/general/extracellular_ephys/NLX-microwires-47 GroupBuilder {'attributes': {'description': 'Microwire', 'location': 'pre_supplementary_motor_area_right', 'namespace': 'core', 'neurodata_type': 'ElectrodeGroup', 'object_id': 'b53b15c9-f151-4d11-bdde-7ef9318d478a'}, 'groups': {}, 'datasets': {}, 'links': {'device': root/general/extracellular_ephys/NLX-microwires-47/device LinkBuilder {'builder': root/general/devices/NLX-microwires-47 GroupBuilder {'attributes': {'description': 'Recordings were performed with Macro-Micro Hybrid Depth Electrodes with Behnke Fried/Micro Inner Wire Bundle in which each individual microwire has a diameter of 40 microns. Likwise, each Depth Electrode has 8 microwires.', 'namespace': 'core', 'neurodata_type': 'Device', 'object_id': '7346546c-0729-42b9-9519-cf2a542eb805'}, 'groups': {}, 'datasets': {}, 'links': {}}}}}, 'NLX-microwires-48': root/general/extracellular_ephys/NLX-microwires-48 GroupBuilder {'attributes': {'description': 'Microwire', 'location': 'pre_supplementary_motor_area_right', 'namespace': 'core', 'neurodata_type': 'ElectrodeGroup', 'object_id': '9915b7ea-fdf3-46eb-8352-ee56df0248ca'}, 'groups': {}, 'datasets': {}, 'links': {'device': root/general/extracellular_ephys/NLX-microwires-48/device LinkBuilder {'builder': root/general/devices/NLX-microwires-48 GroupBuilder {'attributes': {'description': 'Recordings were performed with Macro-Micro Hybrid Depth Electrodes with Behnke Fried/Micro Inner Wire Bundle in which each individual microwire has a diameter of 40 microns. Likwise, each Depth Electrode has 8 microwires.', 'namespace': 'core', 'neurodata_type': 'Device', 'object_id': '218d840d-123f-41dd-83f3-61fa363de4b2'}, 'groups': {}, 'datasets': {}, 'links': {}}}}}, 'NLX-microwires-54': root/general/extracellular_ephys/NLX-microwires-54 GroupBuilder {'attributes': {'description': 'Microwire', 'location': 'amygdala_right', 'namespace': 'core', 'neurodata_type': 'ElectrodeGroup', 'object_id': 'a3ebcb3b-397d-4a60-84d5-b480a2c81751'}, 'groups': {}, 'datasets': {}, 'links': {'device': root/general/extracellular_ephys/NLX-microwires-54/device LinkBuilder {'builder': root/general/devices/NLX-microwires-54 GroupBuilder {'attributes': {'description': 'Recordings were performed with Macro-Micro Hybrid Depth Electrodes with Behnke Fried/Micro Inner Wire Bundle in which each individual microwire has a diameter of 40 microns. Likwise, each Depth Electrode has 8 microwires.', 'namespace': 'core', 'neurodata_type': 'Device', 'object_id': '417ba654-5f7e-4a98-b0fe-653c41b702d7'}, 'groups': {}, 'datasets': {}, 'links': {}}}}}, 'NLX-microwires-7': root/general/extracellular_ephys/NLX-microwires-7 GroupBuilder {'attributes': {'description': 'Microwire', 'location': 'dorsal_anterior_cingulate_cortex_left', 'namespace': 'core', 'neurodata_type': 'ElectrodeGroup', 'object_id': 'b56fcae6-ee65-411d-93be-13dfe22d8fec'}, 'groups': {}, 'datasets': {}, 'links': {'device': root/general/extracellular_ephys/NLX-microwires-7/device LinkBuilder {'builder': root/general/devices/NLX-microwires-7 GroupBuilder {'attributes': {'description': 'Recordings were performed with Macro-Micro Hybrid Depth Electrodes with Behnke Fried/Micro Inner Wire Bundle in which each individual microwire has a diameter of 40 microns. Likwise, each Depth Electrode has 8 microwires.', 'namespace': 'core', 'neurodata_type': 'Device', 'object_id': '057a24c3-68d3-4af0-a8a2-7c613ec28b1b'}, 'groups': {}, 'datasets': {}, 'links': {}}}}}, 'electrodes': root/general/extracellular_ephys/electrodes GroupBuilder {'attributes': {'colnames': array(['x', 'y', 'z', 'location', 'filtering', 'group', 'group_name',
       'origChannel'], dtype=object), 'description': 'microwire electrodes table', 'namespace': 'hdmf-common', 'neurodata_type': 'DynamicTable', 'object_id': 'a12df59b-0739-472e-94dd-d5f0e58cf15b'}, 'groups': {}, 'datasets': {'filtering': root/general/extracellular_ephys/electrodes/filtering DatasetBuilder {'attributes': {'description': 'my description', 'namespace': 'hdmf-common', 'neurodata_type': 'VectorData', 'object_id': '120d4118-3ed9-46bf-b150-2facc7683f76'}, 'data': <StrDataset for Closed HDF5 dataset>}, 'group': root/general/extracellular_ephys/electrodes/group DatasetBuilder {'attributes': {'description': 'my description', 'namespace': 'hdmf-common', 'neurodata_type': 'VectorData', 'object_id': '581cc7dd-dec2-4999-9243-f843bb97fb47'}, 'data': <hdmf.backends.hdf5.h5_utils.BuilderH5ReferenceDataset object at 0x000001EDC8CDE510>}, 'group_name': root/general/extracellular_ephys/electrodes/group_name DatasetBuilder {'attributes': {'description': 'my description', 'namespace': 'hdmf-common', 'neurodata_type': 'VectorData', 'object_id': 'b1fd78ed-f94c-4230-95c2-02c7880ce7e8'}, 'data': <StrDataset for Closed HDF5 dataset>}, 'id': root/general/extracellular_ephys/electrodes/id DatasetBuilder {'attributes': {'namespace': 'hdmf-common', 'neurodata_type': 'ElementIdentifiers', 'object_id': 'e825fd51-5921-4c4e-956e-45d6f1ac89dd'}, 'data': <Closed HDF5 dataset>}, 'location': root/general/extracellular_ephys/electrodes/location DatasetBuilder {'attributes': {'description': 'my description', 'namespace': 'hdmf-common', 'neurodata_type': 'VectorData', 'object_id': '01380354-ff2c-4bf4-88ab-6f6d386ad13d'}, 'data': <StrDataset for Closed HDF5 dataset>}, 'origChannel': root/general/extracellular_ephys/electrodes/origChannel DatasetBuilder {'attributes': {'description': 'my description', 'namespace': 'hdmf-common', 'neurodata_type': 'VectorData', 'object_id': 'a463ee50-9042-4f0d-8f81-f03ff3e36622'}, 'data': <Closed HDF5 dataset>}, 'x': root/general/extracellular_ephys/electrodes/x DatasetBuilder {'attributes': {'description': 'my description', 'namespace': 'hdmf-common', 'neurodata_type': 'VectorData', 'object_id': '500d1a73-00ac-45c4-9c98-7fb929009a33'}, 'data': <Closed HDF5 dataset>}, 'y': root/general/extracellular_ephys/electrodes/y DatasetBuilder {'attributes': {'description': 'my description', 'namespace': 'hdmf-common', 'neurodata_type': 'VectorData', 'object_id': '21dd2272-8c22-4fc1-a9c0-27327e53c5a7'}, 'data': <Closed HDF5 dataset>}, 'z': root/general/extracellular_ephys/electrodes/z DatasetBuilder {'attributes': {'description': 'my description', 'namespace': 'hdmf-common', 'neurodata_type': 'VectorData', 'object_id': '5f8f5266-f264-4f7b-874a-3e087fb70a9e'}, 'data': <Closed HDF5 dataset>}}, 'links': {}}}, 'datasets': {}, 'links': {}}, 'intracellular_ephys': root/general/intracellular_ephys GroupBuilder {'attributes': {}, 'groups': {}, 'datasets': {}, 'links': {}}, 'optogenetics': root/general/optogenetics GroupBuilder {'attributes': {}, 'groups': {}, 'datasets': {}, 'links': {}}, 'optophysiology': root/general/optophysiology GroupBuilder {'attributes': {}, 'groups': {}, 'datasets': {}, 'links': {}}, 'subject': root/general/subject GroupBuilder {'attributes': {'namespace': 'core', 'neurodata_type': 'Subject', 'object_id': '5be8a49f-90ba-4677-ba20-47dcad64d98b'}, 'groups': {}, 'datasets': {'age': root/general/subject/age DatasetBuilder {'attributes': {'reference': 'birth'}, 'data': 'P24Y'}, 'description': root/general/subject/description DatasetBuilder {'attributes': {}, 'data': 'Subject metadata'}, 'sex': root/general/subject/sex DatasetBuilder {'attributes': {}, 'data': 'F'}, 'species': root/general/subject/species DatasetBuilder {'attributes': {}, 'data': 'Homo sapiens'}, 'subject_id': root/general/subject/subject_id DatasetBuilder {'attributes': {}, 'data': '20'}}, 'links': {}}}, 'datasets': {'experiment_description': root/general/experiment_description DatasetBuilder {'attributes': {}, 'data': 'This data contains electrophysiological recordings and behavior from the Sternberg task performed in patients with intractable epilepsy implanted with depth electrodes and Behnke-Fried microwires in the human medial temporal lobe and medial frontal cortex.'}, 'experimenter': root/general/experimenter DatasetBuilder {'attributes': {}, 'data': <StrDataset for Closed HDF5 dataset>}, 'institution': root/general/institution DatasetBuilder {'attributes': {}, 'data': 'Cedars-Sinai Medical Center'}, 'keywords': root/general/keywords DatasetBuilder {'attributes': {}, 'data': <StrDataset for Closed HDF5 dataset>}, 'lab': root/general/lab DatasetBuilder {'attributes': {}, 'data': 'Rutishauser'}, 'notes': root/general/notes DatasetBuilder {'attributes': {}, 'data': 'The session start time has been set to Jan 1st of the recording year to avoid disclosure of protected health information (PHI).'}, 'related_publications': root/general/related_publications DatasetBuilder {'attributes': {}, 'data': <StrDataset for Closed HDF5 dataset>}, 'session_id': root/general/session_id DatasetBuilder {'attributes': {}, 'data': '2'}, 'source_script': root/general/source_script DatasetBuilder {'attributes': {'file_name': 'NWB-SB'}, 'data': 'SB_NWB_export_main.m'}}, 'links': {}}, 'intervals': root/intervals GroupBuilder {'attributes': {}, 'groups': {'trials': root/intervals/trials GroupBuilder {'attributes': {'colnames': array(['loads', 'loadsEnc1_PicIDs', 'loadsEnc2_PicIDs',
       'loadsEnc3_PicIDs', 'loadsProbe_PicIDs', 'start_time', 'stop_time',
       'timestamps_FixationCross', 'timestamps_Encoding1',
       'timestamps_Encoding1_end', 'timestamps_Encoding2',
       'timestamps_Encoding2_end', 'timestamps_Encoding3',
       'timestamps_Encoding3_end', 'timestamps_Maintenance',
       'timestamps_Probe', 'timestamps_Response', 'response_accuracy',
       'probe_in_out'], dtype=object), 'description': 'Intervals for the Sternberg Task', 'namespace': 'core', 'neurodata_type': 'TimeIntervals', 'object_id': 'b6a7b78a-0965-4473-8589-316269b0a338'}, 'groups': {}, 'datasets': {'id': root/intervals/trials/id DatasetBuilder {'attributes': {'namespace': 'hdmf-common', 'neurodata_type': 'ElementIdentifiers', 'object_id': '8f95054f-c60b-44b2-976c-0e056cc55fff'}, 'data': <Closed HDF5 dataset>}, 'loads': root/intervals/trials/loads DatasetBuilder {'attributes': {'description': 'Encoding loads for each trial', 'namespace': 'hdmf-common', 'neurodata_type': 'VectorData', 'object_id': '007ed1dc-0fc2-4899-bd75-8fb059546d3c'}, 'data': <Closed HDF5 dataset>}, 'loadsEnc1_PicIDs': root/intervals/trials/loadsEnc1_PicIDs DatasetBuilder {'attributes': {'description': 'Picture ID for Enc1 loads.', 'namespace': 'hdmf-common', 'neurodata_type': 'VectorData', 'object_id': '1ee3a8bc-f651-423f-9dba-bbd9fd9ee247'}, 'data': <Closed HDF5 dataset>}, 'loadsEnc2_PicIDs': root/intervals/trials/loadsEnc2_PicIDs DatasetBuilder {'attributes': {'description': 'Picture ID for Enc2 loads.', 'namespace': 'hdmf-common', 'neurodata_type': 'VectorData', 'object_id': '720cf055-ce1d-42f0-9939-fc1443b7222a'}, 'data': <Closed HDF5 dataset>}, 'loadsEnc3_PicIDs': root/intervals/trials/loadsEnc3_PicIDs DatasetBuilder {'attributes': {'description': 'Picture ID for Enc1 loads.', 'namespace': 'hdmf-common', 'neurodata_type': 'VectorData', 'object_id': '44bbe04f-9d17-4b0e-ab4e-01035048d30c'}, 'data': <Closed HDF5 dataset>}, 'loadsProbe_PicIDs': root/intervals/trials/loadsProbe_PicIDs DatasetBuilder {'attributes': {'description': 'Picture ID for Probe loads.', 'namespace': 'hdmf-common', 'neurodata_type': 'VectorData', 'object_id': '9e77575f-0141-4552-83ca-ccf777dd766e'}, 'data': <Closed HDF5 dataset>}, 'probe_in_out': root/intervals/trials/probe_in_out DatasetBuilder {'attributes': {'description': 'Whether the probe image was held (1) or not held (0) in memory.', 'namespace': 'hdmf-common', 'neurodata_type': 'VectorData', 'object_id': 'cd515cb1-46a6-4171-aa48-a6742a94acb3'}, 'data': <Closed HDF5 dataset>}, 'response_accuracy': root/intervals/trials/response_accuracy DatasetBuilder {'attributes': {'description': 'Whether the subject response was correct (1) or incorrect (0).', 'namespace': 'hdmf-common', 'neurodata_type': 'VectorData', 'object_id': '765808b0-75cc-4f40-a6f5-a1546593233d'}, 'data': <Closed HDF5 dataset>}, 'start_time': root/intervals/trials/start_time DatasetBuilder {'attributes': {'description': 'Trial start times', 'namespace': 'hdmf-common', 'neurodata_type': 'VectorData', 'object_id': '4bd1076b-eb9f-493f-93c1-7b474c414d3b'}, 'data': <Closed HDF5 dataset>}, 'stop_time': root/intervals/trials/stop_time DatasetBuilder {'attributes': {'description': 'Trial stop times', 'namespace': 'hdmf-common', 'neurodata_type': 'VectorData', 'object_id': '062c57de-ad70-4a85-ab9d-5cabb9c03eb5'}, 'data': <Closed HDF5 dataset>}, 'timestamps_Encoding1': root/intervals/trials/timestamps_Encoding1 DatasetBuilder {'attributes': {'description': 'Start times of picture #1 presentation', 'namespace': 'hdmf-common', 'neurodata_type': 'VectorData', 'object_id': '77aa49f9-9cbd-42d4-93d6-43efcf214ad8'}, 'data': <Closed HDF5 dataset>}, 'timestamps_Encoding1_end': root/intervals/trials/timestamps_Encoding1_end DatasetBuilder {'attributes': {'description': 'End times of picture #1 presentation', 'namespace': 'hdmf-common', 'neurodata_type': 'VectorData', 'object_id': '139b218a-93c6-4efc-a0e4-ae3d6072bc75'}, 'data': <Closed HDF5 dataset>}, 'timestamps_Encoding2': root/intervals/trials/timestamps_Encoding2 DatasetBuilder {'attributes': {'description': 'Start times of picture #2 presentation', 'namespace': 'hdmf-common', 'neurodata_type': 'VectorData', 'object_id': '15e6eea9-23dd-4afd-a046-cb853f0222bb'}, 'data': <Closed HDF5 dataset>}, 'timestamps_Encoding2_end': root/intervals/trials/timestamps_Encoding2_end DatasetBuilder {'attributes': {'description': 'End times of picture #2 presentation', 'namespace': 'hdmf-common', 'neurodata_type': 'VectorData', 'object_id': '543cfcdc-2054-4799-8f34-33569e864ee2'}, 'data': <Closed HDF5 dataset>}, 'timestamps_Encoding3': root/intervals/trials/timestamps_Encoding3 DatasetBuilder {'attributes': {'description': 'Start times of picture #3 presentation', 'namespace': 'hdmf-common', 'neurodata_type': 'VectorData', 'object_id': '15b6b0ad-8804-421d-91bd-3842f25c2f35'}, 'data': <Closed HDF5 dataset>}, 'timestamps_Encoding3_end': root/intervals/trials/timestamps_Encoding3_end DatasetBuilder {'attributes': {'description': 'End times of picture #3 presentation', 'namespace': 'hdmf-common', 'neurodata_type': 'VectorData', 'object_id': '998a08ad-2aca-4638-8754-b3fc59c02734'}, 'data': <Closed HDF5 dataset>}, 'timestamps_FixationCross': root/intervals/trials/timestamps_FixationCross DatasetBuilder {'attributes': {'description': 'Start times of fixation cross', 'namespace': 'hdmf-common', 'neurodata_type': 'VectorData', 'object_id': 'c7f7c5ce-8a8d-42e9-82e0-9aba109cdd67'}, 'data': <Closed HDF5 dataset>}, 'timestamps_Maintenance': root/intervals/trials/timestamps_Maintenance DatasetBuilder {'attributes': {'description': 'Start times of maintenance periods', 'namespace': 'hdmf-common', 'neurodata_type': 'VectorData', 'object_id': 'ab43922f-aabe-4de0-aa74-cca6eff1db14'}, 'data': <Closed HDF5 dataset>}, 'timestamps_Probe': root/intervals/trials/timestamps_Probe DatasetBuilder {'attributes': {'description': 'Start times of probe onset', 'namespace': 'hdmf-common', 'neurodata_type': 'VectorData', 'object_id': '21895212-7d34-4f09-9ee5-1c50d6290864'}, 'data': <Closed HDF5 dataset>}, 'timestamps_Response': root/intervals/trials/timestamps_Response DatasetBuilder {'attributes': {'description': 'Time stamps of button press', 'namespace': 'hdmf-common', 'neurodata_type': 'VectorData', 'object_id': '7f043e1d-cafd-4565-b099-1bec6d256acf'}, 'data': <Closed HDF5 dataset>}}, 'links': {}}}, 'datasets': {}, 'links': {}}, 'processing': root/processing GroupBuilder {'attributes': {}, 'groups': {}, 'datasets': {}, 'links': {}}, 'scratch': root/scratch GroupBuilder {'attributes': {}, 'groups': {}, 'datasets': {}, 'links': {}}, 'stimulus': root/stimulus GroupBuilder {'attributes': {}, 'groups': {'presentation': root/stimulus/presentation GroupBuilder {'attributes': {}, 'groups': {'StimulusPresentation': root/stimulus/presentation/StimulusPresentation GroupBuilder {'attributes': {'comments': 'no comments', 'description': "Presentation order of the stimulus. Indexes 'StimulusTemplates'.", 'namespace': 'core', 'neurodata_type': 'IndexSeries', 'object_id': 'bccf9698-7293-4725-84d0-da2eb9c9dcf8'}, 'groups': {}, 'datasets': {'data': root/stimulus/presentation/StimulusPresentation/data DatasetBuilder {'attributes': {'unit': 'N/A'}, 'data': <Closed HDF5 dataset>}, 'timestamps': root/stimulus/presentation/StimulusPresentation/timestamps DatasetBuilder {'attributes': {'interval': 1, 'unit': 'seconds'}, 'data': <Closed HDF5 dataset>}}, 'links': {'indexed_images': root/stimulus/presentation/StimulusPresentation/indexed_images LinkBuilder {'builder': root/stimulus/templates/StimulusTemplates GroupBuilder {'attributes': {'description': 'A collection of images presented to the subject', 'namespace': 'core', 'neurodata_type': 'Images', 'object_id': 'abf3069d-b04e-4e80-9561-974c5bd9ad58'}, 'groups': {}, 'datasets': {'image_34': root/stimulus/templates/StimulusTemplates/image_34 DatasetBuilder {'attributes': {'description': 'image_34', 'namespace': 'core', 'neurodata_type': 'RGBImage', 'object_id': '5f791e9c-e92d-4376-87cf-0c8085d656b3'}, 'data': <Closed HDF5 dataset>}, 'image_36': root/stimulus/templates/StimulusTemplates/image_36 DatasetBuilder {'attributes': {'description': 'image_36', 'namespace': 'core', 'neurodata_type': 'RGBImage', 'object_id': '86f83808-4352-4f5c-a161-ac7f9a07b80d'}, 'data': <Closed HDF5 dataset>}, 'image_37': root/stimulus/templates/StimulusTemplates/image_37 DatasetBuilder {'attributes': {'description': 'image_37', 'namespace': 'core', 'neurodata_type': 'RGBImage', 'object_id': 'fb6667ee-1e23-47d7-be39-d6f7739a3cd0'}, 'data': <Closed HDF5 dataset>}, 'image_64': root/stimulus/templates/StimulusTemplates/image_64 DatasetBuilder {'attributes': {'description': 'image_64', 'namespace': 'core', 'neurodata_type': 'RGBImage', 'object_id': '8f64cb60-42d9-4503-b53f-9c6494cf24a1'}, 'data': <Closed HDF5 dataset>}, 'image_68': root/stimulus/templates/StimulusTemplates/image_68 DatasetBuilder {'attributes': {'description': 'image_68', 'namespace': 'core', 'neurodata_type': 'RGBImage', 'object_id': 'a4fa50a2-5574-4269-b6b4-f4b68244226d'}, 'data': <Closed HDF5 dataset>}, 'image_999': root/stimulus/templates/StimulusTemplates/image_999 DatasetBuilder {'attributes': {'description': 'image_999', 'namespace': 'core', 'neurodata_type': 'RGBImage', 'object_id': 'c6490443-c14b-47fb-9c95-5d356ffac505'}, 'data': <Closed HDF5 dataset>}, 'order_of_images': root/stimulus/templates/StimulusTemplates/order_of_images DatasetBuilder {'attributes': {'namespace': 'core', 'neurodata_type': 'ImageReferences', 'object_id': 'd0ac853a-e11b-48e3-a3e8-d1c84ee55d84'}, 'data': <hdmf.backends.hdf5.h5_utils.BuilderH5ReferenceDataset object at 0x000001EDC8F15390>}}, 'links': {}}}}}}, 'datasets': {}, 'links': {}}, 'templates': root/stimulus/templates GroupBuilder {'attributes': {}, 'groups': {'StimulusTemplates': root/stimulus/templates/StimulusTemplates GroupBuilder {'attributes': {'description': 'A collection of images presented to the subject', 'namespace': 'core', 'neurodata_type': 'Images', 'object_id': 'abf3069d-b04e-4e80-9561-974c5bd9ad58'}, 'groups': {}, 'datasets': {'image_34': root/stimulus/templates/StimulusTemplates/image_34 DatasetBuilder {'attributes': {'description': 'image_34', 'namespace': 'core', 'neurodata_type': 'RGBImage', 'object_id': '5f791e9c-e92d-4376-87cf-0c8085d656b3'}, 'data': <Closed HDF5 dataset>}, 'image_36': root/stimulus/templates/StimulusTemplates/image_36 DatasetBuilder {'attributes': {'description': 'image_36', 'namespace': 'core', 'neurodata_type': 'RGBImage', 'object_id': '86f83808-4352-4f5c-a161-ac7f9a07b80d'}, 'data': <Closed HDF5 dataset>}, 'image_37': root/stimulus/templates/StimulusTemplates/image_37 DatasetBuilder {'attributes': {'description': 'image_37', 'namespace': 'core', 'neurodata_type': 'RGBImage', 'object_id': 'fb6667ee-1e23-47d7-be39-d6f7739a3cd0'}, 'data': <Closed HDF5 dataset>}, 'image_64': root/stimulus/templates/StimulusTemplates/image_64 DatasetBuilder {'attributes': {'description': 'image_64', 'namespace': 'core', 'neurodata_type': 'RGBImage', 'object_id': '8f64cb60-42d9-4503-b53f-9c6494cf24a1'}, 'data': <Closed HDF5 dataset>}, 'image_68': root/stimulus/templates/StimulusTemplates/image_68 DatasetBuilder {'attributes': {'description': 'image_68', 'namespace': 'core', 'neurodata_type': 'RGBImage', 'object_id': 'a4fa50a2-5574-4269-b6b4-f4b68244226d'}, 'data': <Closed HDF5 dataset>}, 'image_999': root/stimulus/templates/StimulusTemplates/image_999 DatasetBuilder {'attributes': {'description': 'image_999', 'namespace': 'core', 'neurodata_type': 'RGBImage', 'object_id': 'c6490443-c14b-47fb-9c95-5d356ffac505'}, 'data': <Closed HDF5 dataset>}, 'order_of_images': root/stimulus/templates/StimulusTemplates/order_of_images DatasetBuilder {'attributes': {'namespace': 'core', 'neurodata_type': 'ImageReferences', 'object_id': 'd0ac853a-e11b-48e3-a3e8-d1c84ee55d84'}, 'data': <hdmf.backends.hdf5.h5_utils.BuilderH5ReferenceDataset object at 0x000001EDC8F15390>}}, 'links': {}}}, 'datasets': {}, 'links': {}}}, 'datasets': {}, 'links': {}}, 'units': root/units GroupBuilder {'attributes': {'colnames': array(['spike_times', 'electrodes', 'clusterID_orig',
       'waveforms_mean_snr', 'waveforms_peak_snr',
       'waveforms_isolation_distance', 'waveforms_mean_proj_dist'],
      dtype=object), 'description': 'units table', 'namespace': 'core', 'neurodata_type': 'Units', 'object_id': 'c84f66d2-7644-4103-849a-bf4b5e8fac3e'}, 'groups': {}, 'datasets': {'clusterID_orig': root/units/clusterID_orig DatasetBuilder {'attributes': {'description': 'Cluster IDs of units, which referneces the cluster ID used in the native dataset. Used for cross-referencing validating the exported dataset', 'namespace': 'hdmf-common', 'neurodata_type': 'VectorData', 'object_id': '64dffd1f-9570-4b58-a1c0-5682a3c2f3b7'}, 'data': <Closed HDF5 dataset>}, 'electrodes': root/units/electrodes DatasetBuilder {'attributes': {'description': 'single electrodes', 'namespace': 'hdmf-common', 'neurodata_type': 'DynamicTableRegion', 'object_id': '44271797-3f9c-49a6-b425-01a3198cd258', 'table': root/general/extracellular_ephys/electrodes GroupBuilder {'attributes': {'colnames': array(['x', 'y', 'z', 'location', 'filtering', 'group', 'group_name',
       'origChannel'], dtype=object), 'description': 'microwire electrodes table', 'namespace': 'hdmf-common', 'neurodata_type': 'DynamicTable', 'object_id': 'a12df59b-0739-472e-94dd-d5f0e58cf15b'}, 'groups': {}, 'datasets': {'filtering': root/general/extracellular_ephys/electrodes/filtering DatasetBuilder {'attributes': {'description': 'my description', 'namespace': 'hdmf-common', 'neurodata_type': 'VectorData', 'object_id': '120d4118-3ed9-46bf-b150-2facc7683f76'}, 'data': <StrDataset for Closed HDF5 dataset>}, 'group': root/general/extracellular_ephys/electrodes/group DatasetBuilder {'attributes': {'description': 'my description', 'namespace': 'hdmf-common', 'neurodata_type': 'VectorData', 'object_id': '581cc7dd-dec2-4999-9243-f843bb97fb47'}, 'data': <hdmf.backends.hdf5.h5_utils.BuilderH5ReferenceDataset object at 0x000001EDC8CDE510>}, 'group_name': root/general/extracellular_ephys/electrodes/group_name DatasetBuilder {'attributes': {'description': 'my description', 'namespace': 'hdmf-common', 'neurodata_type': 'VectorData', 'object_id': 'b1fd78ed-f94c-4230-95c2-02c7880ce7e8'}, 'data': <StrDataset for Closed HDF5 dataset>}, 'id': root/general/extracellular_ephys/electrodes/id DatasetBuilder {'attributes': {'namespace': 'hdmf-common', 'neurodata_type': 'ElementIdentifiers', 'object_id': 'e825fd51-5921-4c4e-956e-45d6f1ac89dd'}, 'data': <Closed HDF5 dataset>}, 'location': root/general/extracellular_ephys/electrodes/location DatasetBuilder {'attributes': {'description': 'my description', 'namespace': 'hdmf-common', 'neurodata_type': 'VectorData', 'object_id': '01380354-ff2c-4bf4-88ab-6f6d386ad13d'}, 'data': <StrDataset for Closed HDF5 dataset>}, 'origChannel': root/general/extracellular_ephys/electrodes/origChannel DatasetBuilder {'attributes': {'description': 'my description', 'namespace': 'hdmf-common', 'neurodata_type': 'VectorData', 'object_id': 'a463ee50-9042-4f0d-8f81-f03ff3e36622'}, 'data': <Closed HDF5 dataset>}, 'x': root/general/extracellular_ephys/electrodes/x DatasetBuilder {'attributes': {'description': 'my description', 'namespace': 'hdmf-common', 'neurodata_type': 'VectorData', 'object_id': '500d1a73-00ac-45c4-9c98-7fb929009a33'}, 'data': <Closed HDF5 dataset>}, 'y': root/general/extracellular_ephys/electrodes/y DatasetBuilder {'attributes': {'description': 'my description', 'namespace': 'hdmf-common', 'neurodata_type': 'VectorData', 'object_id': '21dd2272-8c22-4fc1-a9c0-27327e53c5a7'}, 'data': <Closed HDF5 dataset>}, 'z': root/general/extracellular_ephys/electrodes/z DatasetBuilder {'attributes': {'description': 'my description', 'namespace': 'hdmf-common', 'neurodata_type': 'VectorData', 'object_id': '5f8f5266-f264-4f7b-874a-3e087fb70a9e'}, 'data': <Closed HDF5 dataset>}}, 'links': {}}}, 'data': <Closed HDF5 dataset>}, 'id': root/units/id DatasetBuilder {'attributes': {'namespace': 'hdmf-common', 'neurodata_type': 'ElementIdentifiers', 'object_id': '7730afdf-e871-4c37-8b5b-a758fcc5b720'}, 'data': <Closed HDF5 dataset>}, 'spike_times': root/units/spike_times DatasetBuilder {'attributes': {'description': 'Timestamps when spikes occured (seconds)', 'namespace': 'hdmf-common', 'neurodata_type': 'VectorData', 'object_id': 'ffe0db87-9ca4-4b3d-bdbd-7b4d0786cc80'}, 'data': <Closed HDF5 dataset>}, 'spike_times_index': root/units/spike_times_index DatasetBuilder {'attributes': {'description': 'indexes data', 'namespace': 'hdmf-common', 'neurodata_type': 'VectorIndex', 'object_id': '8585ba4d-8a6d-43f5-b6cd-94992e7d721e', 'target': root/units/spike_times DatasetBuilder {'attributes': {'description': 'Timestamps when spikes occured (seconds)', 'namespace': 'hdmf-common', 'neurodata_type': 'VectorData', 'object_id': 'ffe0db87-9ca4-4b3d-bdbd-7b4d0786cc80'}, 'data': <Closed HDF5 dataset>}}, 'data': <Closed HDF5 dataset>}, 'waveform_mean': root/units/waveform_mean DatasetBuilder {'attributes': {'description': 'Mean waveform for each unit.', 'namespace': 'hdmf-common', 'neurodata_type': 'VectorData', 'object_id': '1ae5bf58-b506-4749-9060-f65cb1972761', 'sampling_rate': 100000.0, 'unit': 'microvolts'}, 'data': <Closed HDF5 dataset>}, 'waveform_sd': root/units/waveform_sd DatasetBuilder {'attributes': {'description': 'Standard deviation for all waveform means at each timestamp.', 'namespace': 'hdmf-common', 'neurodata_type': 'VectorData', 'object_id': 'e136f277-480e-4bca-929e-8ed7fd684644', 'sampling_rate': 100000.0, 'unit': 'microvolts'}, 'data': <Closed HDF5 dataset>}, 'waveforms': root/units/waveforms DatasetBuilder {'attributes': {'description': 'Array of Nx256 waveforms across all clusters.', 'namespace': 'hdmf-common', 'neurodata_type': 'VectorData', 'object_id': '8753a19f-b8d3-4e09-819d-4a8f6ad12948', 'sampling_rate': 100000.0, 'unit': 'microvolts'}, 'data': <Closed HDF5 dataset>}, 'waveforms_index': root/units/waveforms_index DatasetBuilder {'attributes': {'description': 'indexes data', 'namespace': 'hdmf-common', 'neurodata_type': 'VectorIndex', 'object_id': '99dfdbe0-a068-4763-9969-d463e9695c47', 'target': root/units/waveforms DatasetBuilder {'attributes': {'description': 'Array of Nx256 waveforms across all clusters.', 'namespace': 'hdmf-common', 'neurodata_type': 'VectorData', 'object_id': '8753a19f-b8d3-4e09-819d-4a8f6ad12948', 'sampling_rate': 100000.0, 'unit': 'microvolts'}, 'data': <Closed HDF5 dataset>}}, 'data': <Closed HDF5 dataset>}, 'waveforms_index_index': root/units/waveforms_index_index DatasetBuilder {'attributes': {'description': 'indexes indexes', 'namespace': 'hdmf-common', 'neurodata_type': 'VectorIndex', 'object_id': 'ea440ec5-0af8-40e2-946c-08d160bd7cf8', 'target': root/units/waveforms_index DatasetBuilder {'attributes': {'description': 'indexes data', 'namespace': 'hdmf-common', 'neurodata_type': 'VectorIndex', 'object_id': '99dfdbe0-a068-4763-9969-d463e9695c47', 'target': root/units/waveforms DatasetBuilder {'attributes': {'description': 'Array of Nx256 waveforms across all clusters.', 'namespace': 'hdmf-common', 'neurodata_type': 'VectorData', 'object_id': '8753a19f-b8d3-4e09-819d-4a8f6ad12948', 'sampling_rate': 100000.0, 'unit': 'microvolts'}, 'data': <Closed HDF5 dataset>}}, 'data': <Closed HDF5 dataset>}}, 'data': <Closed HDF5 dataset>}, 'waveforms_isolation_distance': root/units/waveforms_isolation_distance DatasetBuilder {'attributes': {'description': 'Cluster Isolation distance, computed using all waveforms in the cluster.', 'namespace': 'hdmf-common', 'neurodata_type': 'VectorData', 'object_id': 'fc0148f3-30d5-4bb1-9724-02f30b921fdf'}, 'data': <Closed HDF5 dataset>}, 'waveforms_mean_proj_dist': root/units/waveforms_mean_proj_dist DatasetBuilder {'attributes': {'description': 'Cluster Isolation distance, computed using all waveforms in the cluster.', 'namespace': 'hdmf-common', 'neurodata_type': 'VectorData', 'object_id': 'd3f3519d-1742-49d2-81a1-d3ada7e7f473'}, 'data': <Closed HDF5 dataset>}, 'waveforms_mean_snr': root/units/waveforms_mean_snr DatasetBuilder {'attributes': {'description': 'Mean Signal-to-Noise Ratio (SNR) across all samples of the mean waveform.', 'namespace': 'hdmf-common', 'neurodata_type': 'VectorData', 'object_id': '4f683fc5-4ad2-4acd-996c-f8b70b13985e'}, 'data': <Closed HDF5 dataset>}, 'waveforms_peak_snr': root/units/waveforms_peak_snr DatasetBuilder {'attributes': {'description': 'Signal-to-Noise Ratio (SNR) of the mean signal amplitude.', 'namespace': 'hdmf-common', 'neurodata_type': 'VectorData', 'object_id': '82fa1cb8-ac95-4147-adb1-fb823ed45bb3'}, 'data': <Closed HDF5 dataset>}}, 'links': {}}}, 'datasets': {'file_create_date': root/file_create_date DatasetBuilder {'attributes': {}, 'data': <StrDataset for Closed HDF5 dataset>}, 'identifier': root/identifier DatasetBuilder {'attributes': {}, 'data': 'SBID_20_P49CS'}, 'session_description': root/session_description DatasetBuilder {'attributes': {}, 'data': 'SBID: 20'}, 'session_start_time': root/session_start_time DatasetBuilder {'attributes': {}, 'data': '2017-01-01'}, 'timestamps_reference_time': root/timestamps_reference_time DatasetBuilder {'attributes': {}, 'data': '2017-01-01'}}, 'links': {}}, "Could not construct NWBFile object due to: 'timestamps_reference_time' must be a timezone-aware datetime object.")

Operating System

Windows

Matlab Version

2023a

Code of Conduct

lawrence-mbf commented 5 months ago

By newest do you mean the newest release or the tip of master? I would recommend the tip of master.

Also, as mentioned in the error, you will need to set the relevant time zone using the TimeZone property in your datetime object. The setter should do this automatically, however.

NeuroGuth commented 5 months ago

Hi Lawrence, thank you very much for your quick reply! I meant the tip of master, I apologize for the confusion. I did set the timezone property, the error remained. I thought, the problem is that I cannot assign any datatime objects to the nwbFile without them being stored in a cell. Or are they supposed to be stored in a cell and then exported as datetime objects anyway? Thanks a lot for your help!

lawrence-mbf commented 5 months ago

I believe them being in cells is intentional but I also don't think that is relevant to the pynwb error.

You may also need to set the time to a non-zero value along with a time zone in order for the format to be correct. The Pynwb error is just complaining that the string format is not iso8601.

NeuroGuth commented 5 months ago

thank you very much! I will try this out tomorrow and let you know of the result!

NeuroGuth commented 5 months ago

Hi Lawrence,

your advice was indeed correct! If, the times are set to non-zero values, everything works fine. I greatly apprechiate your help!

Best, Tim

lawrence-mbf commented 5 months ago

Hi @NeuroGuth ,

Please try not to edit the same post over and over, it makes it difficult to know where the conversation is going.

The main issue appears to be the format detection for datetime objects in this snippet: https://github.com/NeurodataWithoutBorders/matnwb/blob/33b8178ecfa9be632e750becae3b41e94a527e24/%2Btypes/%2Butil/correctType.m#L76-L85

When the special case of time being all zeros occurs, the formatter assumed that the time was invalid and dropped both the time and any relevant time zones. Since this causes errors in pynwb, I have now changed the snippet such that: https://github.com/NeurodataWithoutBorders/matnwb/blob/9d77234352d64bfebe52e103ef6f37aba295e737/%2Btypes/%2Butil/correctType.m#L76-L83

It only checks for timezone and assumes that the time is always present. In fact, there really was no way to know if hours/minutes/seconds were set, just if TimeZone was never provided.

If you can, please pull from master and let me know if this worked for you.

NeuroGuth commented 5 months ago

Hi Lawrence,

great! Thank you very much, that fixes the issue! Maybe one could even consider the following code:

function Datetime = formatDatetime(Datetime)
formatString        = 'yyyy-MM-dd''T''HH:mm:ss.SSSSSSZZZZZ';
if isempty(Datetime.TimeZone)
    localTime           = datetime('now', 'TimeZone', 'local');
    localTime.Format    = formatString;
    Datetime.TimeZone   = localTime.TimeZone;
    warning('no time zone specified - set to local time zone');
end
Datetime.Format = formatString;
end

This might improve compatibility with pyNWB, if someone forgets to add a time zone. It might also be helpful, if pyNWB wouldn't require a time zone to read in the nwb-file.

Thanks a lot Lawrence!