[Bug]: mismatch between timestamps and frames in miniscope video

laurelrr commented 1 month ago

What happened?

I am working with this repo: https://github.com/catalystneuro/tye-lab-to-nwb/tree/main/src/tye_lab_to_nwb/ast_ophys and attempting to convert Miniscope calcium recordings but got the same error on 2 subjects.

Steps to Reproduce

python convert_session.py

Traceback

Traceback (most recent call last):
  File "/home/lkeyes/Projects/GIT/tye-lab-to-nwb/src/tye_lab_to_nwb/ast_ophys/convert_session_carl.py", line 149, in session_to_nwb
    converter.run_conversion(
  File "/nadata/snlkt/home/lkeyes/Projects/GIT/tye-lab-to-nwb/src/tye_lab_to_nwb/ast_ophys/ast_ophysnwbconverter.py", line 59, in run_conversion
    super().run_conversion(
  File "/home/lkeyes/anaconda3/envs/tye_lab_to_nwb_env/lib/python3.11/site-packages/neuroconv/nwbconverter.py", line 160, in run_conversion
    self.add_to_nwbfile(nwbfile_out, metadata, conversion_options)
  File "/home/lkeyes/anaconda3/envs/tye_lab_to_nwb_env/lib/python3.11/site-packages/neuroconv/nwbconverter.py", line 114, in add_to_nwbfile
    data_interface.add_to_nwbfile(
  File "/home/lkeyes/anaconda3/envs/tye_lab_to_nwb_env/lib/python3.11/site-packages/neuroconv/datainterfaces/ophys/miniscope/miniscopeconverter.py", line 64, in add_to_nwbfile
    self.data_interface_objects["MiniscopeImaging"].add_to_nwbfile(
  File "/home/lkeyes/anaconda3/envs/tye_lab_to_nwb_env/lib/python3.11/site-packages/neuroconv/datainterfaces/ophys/miniscope/miniscopeimagingdatainterface.py", line 91, in add_to_nwbfile
    imaging_extractor.set_times(times=miniscope_timestamps)
  File "/home/lkeyes/anaconda3/envs/tye_lab_to_nwb_env/lib/python3.11/site-packages/roiextractors/imagingextractor.py", line 196, in set_times
    assert len(times) == self.get_num_frames(), "'times' should have the same length of the number of frames!"
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'times' should have the same length of the number of frames!

Operating System

Linux

Python Executable

Conda

Python Version

3.11

Package Versions

environment_for_issue.txt

Code of Conduct

Yes

Duplicated Issue Check

Yes

h-mayorquin commented 1 month ago

@weiglszonja. @alessandratrapani (tagging both of you as we have worked on this)

I discussed this with @laurelrr it seems that one of the sessions for the Tye lab has a session where the timestamps do not match the number of frames. So there is an error here:

https://github.com/catalystneuro/roiextractors/blob/2cf9235d9e43c8070744922b4649d7482e99c2ec/src/roiextractors/imagingextractor.py#L189-L198

The timestamps are obtained from a csv that is specific to miniscope and aggregated here:

https://github.com/catalystneuro/neuroconv/blob/52cd6aa0ea46c0d143739767da4951ade17c3b74/src/neuroconv/datainterfaces/ophys/miniscope/miniscopeimagingdatainterface.py#L80-L85

And the frames are the sum of the frames of each individual extractor here:

https://github.com/catalystneuro/roiextractors/blob/2cf9235d9e43c8070744922b4649d7482e99c2ec/src/roiextractors/multiimagingextractor.py#L40-L47

Which in turn are obtained from the video directly using OpenCV.

https://github.com/catalystneuro/roiextractors/blob/2cf9235d9e43c8070744922b4649d7482e99c2ec/src/roiextractors/extractors/miniscopeimagingextractor/miniscopeimagingextractor.py#L105-L107

So I think that either the timestamps in the csv are wrong or there are some drop frames in some videos. @laurelrr will see how he can share the data with us so we can verify that.

Meanwhile, what should we do in the case there are dropped frames? I think dropping frames is somehow common in acquisition systems that involve video so I think we should handle that case gracefully but I am not sure how.

alessandratrapani commented 1 month ago

The miniscope TimeStamps.csv has a dedicated column "Buffer_Index" to signal problems with frame acquisition, e.g., dropped frames. We can start by looking at this piece of data.

h-mayorquin commented 1 month ago

OK, this happens because there are some files that match the *.avi wildcard but not in the intended way in one of the segments:

├── 13_59_04
│   ├── BehavCam_2
│   │   ├── 0.avi
│   │   ├── 1.avi
│   │   ├── 2.avi
│   │   ├── 3.avi
│   │   ├── 4.avi
│   │   ├── 5.avi
│   │   ├── 6.avi
│   │   ├── 7.avi
│   │   ├── 8.avi
│   │   ├── metaData.json
│   │   └── timeStamps.csv
│   ├── metaData.json
│   ├── Miniscope
│   │   ├── 0.avi
│   │   ├── 0_corrected.avi
│   │   ├── 1.avi
│   │   ├── 2.avi
│   │   ├── 3.avi
│   │   ├── 4.avi
│   │   ├── metaData.json
│   │   └── timeStamps.csv
│   └── notes.csv
└── G2B4_C5-6852_L_yeti_DISC6UNP+TONES_msCamAll_ffd.mat

See the 0_corrected.avi

Can/should we improve the extractor to not match things with letters?

laurelrr commented 1 day ago

OK, my fix for this was to make a copy of the data folder and mv 0_corrected.avi 0.avi for each instance of a mislabeled filename.
I assume I did not need to reprocess any of the other downstream *.mat files that are used in convert_session.py, but please correct me if that seems wrong. I'm trying it out now and will let you know the results.

laurelrr commented 1 day ago

OK, my idea did not seem to fix the issue. I will confer with my lab mates and see what else might have been affected.

laurelrr commented 5 hours ago

After conferring with the team, it looks like we need only the 0.avi behavCam and the 0_corrected.avi miniscope file (which I have renamed to 0.avi). I set this up in my copy of the original folder. it now looks like this:

├── 13_59_04
│   ├── BehavCam_2
│   │   ├── 0.avi
│   │   ├── metaData.json
│   │   └── timeStamps.csv
│   ├── metaData.json
│   ├── Miniscope
│   │   ├── 0.avi   (previously, 0_corrected.avi)
│   │   ├── metaData.json
│   │   └── timeStamps.csv
│   └── notes.csv
└── G2B4_C5-6852_L_yeti_DISC6UNP+TONES_msCamAll_ffd.mat

I am trying to run pulling from the updated miniscope folder, but still see the same error message:

Traceback (most recent call last):
  File "/nadata/snlkt/home/lkeyes/Projects/GIT/tye-lab-to-nwb/src/tye_lab_to_nwb/ast_ophys/convert_session_yeti.py", line 149, in session_to_nwb
    converter.run_conversion(
  File "/nadata/snlkt/home/lkeyes/Projects/GIT/tye-lab-to-nwb/src/tye_lab_to_nwb/ast_ophys/ast_ophysnwbconverter.py", line 59, in run_conversion
    super().run_conversion(
  File "/home/lkeyes/anaconda3/envs/tye_lab_to_nwb_env/lib/python3.11/site-packages/neuroconv/nwbconverter.py", line 160, in run_conversion
    self.add_to_nwbfile(nwbfile_out, metadata, conversion_options)
  File "/home/lkeyes/anaconda3/envs/tye_lab_to_nwb_env/lib/python3.11/site-packages/neuroconv/nwbconverter.py", line 114, in add_to_nwbfile
    data_interface.add_to_nwbfile(
  File "/home/lkeyes/anaconda3/envs/tye_lab_to_nwb_env/lib/python3.11/site-packages/neuroconv/datainterfaces/ophys/miniscope/miniscopeconverter.py", line 64, in add_to_nwbfile
    self.data_interface_objects["MiniscopeImaging"].add_to_nwbfile(
  File "/home/lkeyes/anaconda3/envs/tye_lab_to_nwb_env/lib/python3.11/site-packages/neuroconv/datainterfaces/ophys/miniscope/miniscopeimagingdatainterface.py", line 91, in add_to_nwbfile
    imaging_extractor.set_times(times=miniscope_timestamps)
  File "/home/lkeyes/anaconda3/envs/tye_lab_to_nwb_env/lib/python3.11/site-packages/roiextractors/imagingextractor.py", line 196, in set_times
    assert len(times) == self.get_num_frames(), "'times' should have the same length of the number of frames!"
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError: 'times' should have the same length of the number of frames!

Is there something else that should be updated, beyond just deleting extraneous files?

catalystneuro / roiextractors