Debug problems with composite video generation

[anayapouget]

Until recently I was focusing on composite videos of Aeon 3 social 2 which have minimal issues since I believe these are the ones Orsi was testing her code with. However, I started noting some issues when I moved on to Aeon 4 social 02, and then with social 03 and 04 it gets worse... Here's a list of the problems I have noticed so far:

1. Aeon 4 social 03, social period 2024-06-19T14-00-00: there are some frames where no mouse is in frame
2. Aeon 3 social 04, presocial period 2024-08-20T11-00-00 + Aeon 4 social 02, presocial period 2024-01-31_11-00-00 and 2024-02-01T15-00-00: the generated composite videos are shorter than expected, frames must have been dropped
3. Aeon 4 social 04, presocial period 2024-08-16T17-00-00: all frames are sampled from the same quadrant camera

Possible explanations:

• Issues with Bonsai centroid tracking or DJ SLEAP full pose ID data i.e., some frames where it is missing, and/or some frames where it is inaccurate?
• Blind spots i.e., some spots in the arena are perhaps not covered by any of the quad cameras? I think this is unlikely though
• Innacurate homography, though the pixel mapping results plots look pretty decent so I would be surprised this is causing massive issues? Check if it's the case in particular for Aeon 4 social 03, social period 2024-06-19T14-00-00 because there is one camera angle where I am getting weird SLEAP labels (but not in individual sessions which is weird, so maybe the DJ SLEAP full pose ID data is to blame?)
• Bugs in Orsi's code that assigns quadrant camera frames to top camera frames, causing some frames to be dropped or wrong assignments

[anayapouget]

1. Cause of problem: There are discrepancies between top camera and quadrant camera timestamps causing the wrong quadrant camera frames to be pulled. This might be due to my bonsai sleap workflow? Solution: Find the root cause. If it is due to the bonsai SLEAP workflow, on the long term, the timestamps for the DJ top camera position data need to be fixed. On the short term, I can write code to open the CameraTop csv, view the harp timestamps for the beginning and end of the chunk (these are the true timestamps of the DJ position data) and use those to find the correct quadrant camera frames.
2. Cause of problem: some epochs are missing quadrant camera data (i.e., the folder for a quadrant camera is missing). Solution: after running the new quadrant camera bonsai sleap workflow, we will have to run a second piece of code that completes frames missing from the composite videos with CameraTop pose data.
3. Cause of problem: the bonsai centroid tracking is stuck on the top of the nest and not actually tracking the mouse. I believe @NeuroThom made a new bonsai centroid tracking workflow that fixes this issue? Solution: either we run @NeuroThom 's fix on the data, or we wait until I manage to generate SLEAP position data for these chunks. I am currently struggling to do so though because I need to find a way to restrict the number of detected instances when running sleap predictions through Bonsai. @glopesdev do you know if that is possible to do?