Inaccurate sklearn Results

[mrnels19]

Describe the bug In the videos generated by sklearn, the skeletons are not in the same position as the mouse.

To Reproduce Steps to reproduce the behavior:

1. Open SimBA and create a new project. Under SML Settings, set the number of predictive classifiers equal to 3 and give each classifier a name.
2. Select Multi Tracking under the Type of Tracking menu and create a user-defined pose config that is for two animals, each with 13 body parts. Select this user-defined pose config for the project.
3. Click on Generate Pose Config and then advance to the Import videos into project folder tab.
4. Fill in the path information under the Import single video section and press Import videos (I imported two videos. Each had around 74000 frames and the frame rate was 60 frames/second). Then, click on the Import tracking data tab.
5. For File type select SLP (SLEAP), choose Animal(s): Linear for the interpolation and Gaussian for the smoothing. Set the period equal to 100 ms. For number of animals, put 2 and give each animal a label.
6. Extract frames into the project folder.
7. Next, load the project and go to the video parameters tab. Enter in a known distance and complete the appropriate steps.
8. Under the Outlier Correction tab, click Settings and have body part 1 be the snout and body part 2 be the left ear for both the location and movement corrections. Set the location criterion equal to 1 under the location correction, and set the location criterion equal to 2 for the movement correction. For aggregation method, click median. (I am still learning about the parameters I set here, so if these parameters were not optimal, feel free to let me know)
9. Extract Features
10. For labeling behavior, I used the advanced label behavior method. since I had so many frames in each video, I only annotated once every 50 frames for the first 20000 frames in one video for this test.
11. To train the machine model, I used the model parameters provided on the SimBA GitHub, and they can be found in the screenshot attached. I chose the Save settings for specific model option, and I created three models - one for each behavior.
12. Under the Run Machine Model tab, I clicked on all the buttons under the Analyze Machine Results tab except for Classifications by ROI (I did not do ROI for this test)
13. I then clicked on the Visualizations tab and under Sklearn Visualization section, I found the Apply to all videos subsection, where I checked the Generate video box and the button that said Visualize classification results. This was performed on both of the videos I imported. One of the videos is attached below, and the placements of the skeleton are not accurate, even though the SLEAP data on the mice was.

Expected behavior The skeletons on these mice in the sklearn videos should have been accurate. Another thing I want to point out is that when I click on the user-defined pose configuration I created that has 13 markers on each animal, a picture of the mice I labeled in the process of creating this configuration does not appear. Instead, I see the image in the screenshot below. I am not sure why this incorrect picture is appearing. I am not sure if this picture is playing a role.

Screenshots The screenshots and video are attached.

Desktop (please complete the following information):

• Windows
• Python 3.6.13
• I am using Anaconda

[mrnels19]

Here is a photo of an inaccurate frame. The video was too large to upload.

[sronilsson]

Hi @mrnels19!

I can see that the relative distances between the pose-estimated locations and the actual locations appears to be a lot smaller then they should be. This could happen if you performed pose-estimation on, say as example, 200x100 resolution video, then you imported a 1000x500 version of the same video into SimBA.

Next, when you try to plot the results, SimBA would grab the 1000x500 resolution video that you imported, and plot the pose generated from a 200x100 resolution video.

Is it possible this could have happened?

[mrnels19]

Hi @sronilsson! Thank you for your reply! The image that I labeled the body parts involved in my user-defined pose configuration had a width of 2440 pixels and a height of 1800 pixels (it was a screenshot I took of one frame in the video). Each video that I imported had a frame width of 1280 pixels and a height of 1024 pixels. I found this information by looking at the properties of each image/video on my computer, and I also checked the video frame/resolution width and height in SimBA by pressing the Set video parameters button under the Video parameters tab in the Load Project GUI. I am not sure how the resolution of the video could have changed in SimBA.

[mrnels19]

Hi, I just wanted to follow up to my previous comment with a short screen recording where I pass through one of my sklearn video results. In the screen recording, the tracking starts off fine, but then the body parts for both mice get clumped together, and then no body parts are present towards the end of the video. Could this be an issue with the hyperparameters or a lack of behavior annotations when training the model? Or, could it be due to the videos themselves (like a change in resolution) or another issue? Thank you for your time!

Video Link: https://drive.google.com/file/d/1cUIuBv6vc-W8DScjnEbeY6JFQbqlAkJx/view?usp=sharing

[sronilsson]

Hi @mrnels19!

Before troubleshooting, I just want to confirm that the animals in the video are looking as expected? I'm looking at the video and I can see the animals jumping around in the open field from one frame to the next.

The issue is not going to be hyperparameters or annotations, but the pose estimation tracking data, either before it's imported or when modified during smoothing, interpolation, or outlier correction in SimBA.

[mrnels19]

Hello! Thank you so much for your help with this. I was able to resolve the issue by adjusting the smoothing, interpolation, and outlier correction in SimBA.