search

catalystneuro / datta-lab-to-nwb

MIT License
1 stars 2 forks source link

Align Raw Video to TDT timestamps #61

Closed pauladkisson closed 9 months ago

pauladkisson commented 1 year ago

Raw video timestamps are not currently aligned, but Datta Lab has some process to align them with IR LED pulses copied to the TDT.

pauladkisson commented 12 months ago

Based on today's discussion, the alignment code in align.py, and the output .parquet tables (photometry_df.parquet and photometry_df_v2.parquet), I've gained the following insights about the alignment process:

  1. The raw video timestamps (depth_ts.txt) are treated as ground-truth, and the photometry data is simultaneously aligned and downsampled (6kHz --> 30Hz) to these.

  2. The processed timestamps in the .parquet tables (dlight_raw_data/dlight_photometry_processed_full.parquet) are interpolated to account for dropped frames --> this is why the processed data ends up being longer than the original videos. Behavioral syllables are also interpolated, but the kinematic data is filled with NaNs, which seems reasonable.

  3. The alignment process drops a substantial amount of fiber photometry data (~5mins for the example I'm working with) --> these are filled with NaNs in the processed .parquet tables

  4. The processed .parquet table for the example session I am using matches photometry_df.parquet rather than photometry_df_v2.parquet --> I would guess that this is hodgepodge across sessions.

    As a result, I see 2 main problems in the alignment process:

  5. How to align the 6kHz raw FP between the 30Hz aligned points?

  6. How to align the tails not accounted for in the current alignment process?

And I can propose 2 initial (relatively naive) solutions:

  1. Interpolate (linearly?) between aligned points
  2. Extrapolate linearly from the outermost aligned points following the stated fs of the TDT BUT this might convey more certainty in the unaligned tails than we really want...
pauladkisson commented 12 months ago

@wingillis, I would love to hear your thoughts on solutions as well as double-checking my understanding here.

pauladkisson commented 9 months ago

Proposed Solution:

  1. Use the linear fit from the alignment dataframes to get all the 500Hz photometry indices that correspond to the depth video frames
  2. upsample the indices from 500Hz to the raw sampling frequency (~6kHz)
  3. Interpolate in between each sparsely aligned index using linear interpolation
  4. Shift all timestamps by the starting time determined by the 0 index of the depth video.

See example implementation.