sronilsson
commented
1 month ago Hi @SnowySnowySnowy !
Short answer Yes.
TLDR: Preprocessing is for when recordings are made in less-than-optimal conditions for pose-estimation and is used to improve pose-estimation. Better input data gives between classifications. Often parts captured in videos are areas outside arena which animal can never reach, some cable that could be mistaken for a tail is there, or early captured portions can feature the experimenters hands in images etc., mucking up the pose-estimation performance and later classifications. The resolution and FPS may be also unnecessarily high introducing long pose-estimation and classification runtimes and unnecessary large disk space requirements.
Much can be fixed by clipping/cropping/re-sampling the videos, and you want to do this before performing pose-estimation.
If you performed pose-estimation, then resampled the videos, and imported the resampled videos but the original pose-estimation to SimBA, there could be issues in visualization components in SimBA. For example, the nose could be pose-estimated at pixel location (1600, 800). However, your cropped videos are only 800 by 600 large so 1600x600 is outside the image. If you re-sampled the FPS then there would be a similar mixup in time domain so you see the pose-estimation moving much quicker then the actual animal etc.
I have a question regarding the video preprocessing step for simba. Should videos be preprocessed as shown in the simba tutorial, then run through SLEAP, and then also be used in simba? In other words, is the expectation that the processed videos are the ones used for pose estimation and then behavioral classification?