Grid movies for 3D data

Currently generate_grid_movies only works for 2D data because it assumes that keypoint coordinates are points in pixel space. We should address this by creating a new function generate_grid_movies_3D that takes camera calibrations as additional input.

One complication is that for each recording session, there will be multiple video files, and potentially separate calibration files. Furthermore, the format in which these are named may vary from method to method. Therefore, I propose that we make people create a table that explicitly links these together... so each row of the table would contain: 1) the name of the recording session (corresponding to a key in the results.h5 output by keypoint-moseq) 2) the path to a video file 3) the path to a calibration file 4) [optional] the path to a file that contains a list of frames indexes. this would be necessary if some video frames were thrown out during synchronization.

When generating grid movies, the caller should provide this table (or a link to the table), a string specifying which method they used (e.g. "sleap-anipose"), and a flag stating whether the camera they have chosen is a top-down view. Also, if the method is "anipose" or "sleap-anipose", the caller should provide the camera name, since this is needed to read the calibration file.

To write generate_grid_movies_3D, we need to make the following changes from generate_grid_movies 1) Get session/video correspondences from the table, rather than by string matching 2) If the user provided frame indexes, make sure that the total number of such indexes matches the length of the keypoint-MoSeq output for each session. If they didn't, then we should make a warning asking them to supply frame indexes 3) If the user specified that the camera is a top-down view, then we should perform egocentric alignment of the videos as usual. Otherwise, we should do centering but not rotation. 4) Extract calibration parameters. The format of the parameters will differ by method so we might need a few loading functions. 5) Use the calibration params to project 3D coordinates (and centroids) into the 2D image plane. I wrote a util for this in another library: https://github.com/calebweinreb/multicam-calibration/blob/127fba29a0067323a41df1d66235fe0829669fa7/multicam_calibration/geometry.py#L147

In terms of documenting this, I think it would be nice to make a separate tutorial notebook that describes data loading and visualization for people who have 3D data.

dattalab / keypoint-moseq

Grid movies for 3D data #64