Filter out low-scoring skeletons spawned during inference

[olinesn]

Problem: We only allow for filtering based on instance score during tracking. We should have options for filtering even without tracking (e.g., during the HITL labeling process).

Solution: Add Predictor options and associated CLI flags and GUI fields to allow for filtering by score during inference independently of tracking.

This should also allow for filtering based on number of animals in addition to an absolute score threshold. This would cover the use case of doing single-animal tracking using multi-animal models.

See #640 for the first point related to including background frames.

---

Original issue:

As far as I understand, only labelled frames with skeletons are included in the training set. I have a prep where I occasionally have to reach my hand into the cage while the camera is on, and SLEAP inference ends up spawning lots of tracks that are low-scoring skeletons on my glove/arm.

What's the best way to deal with this? Options I can think of include:

1: Is there a way to include frames in the training set that do NOT have any animals in them, to teach the model what's NOT an animal?

2: Can I set a threshold for skeleton score when running inference on a video, and remove all skeletons below a threshold BEFORE they get stitched into tracks?

Thanks!

[olinesn]

This could potentially merged with #640

[talmo]

Hi @olinesn,

1. I don't remember if we tested this, but one workaround could be to have a skeleton with all nodes marked as not visible. I think this might be sufficient to have the frame included as a "negative" example. (This will not work, see comment below)

2. Not quite the same, but using the sleap-track options --tracking.target_instance_count 3 --tracking.pre_cull_to_target 1 without setting the IOU will filter instances by score before tracking them. Not ideal though as it can still include spurious instances as long as it's less than the target count.

We'll work on these as enhancements. Thanks!!

Talmo

[roomrys]

Since we seem to have gotten a lot of cases where people are marking all nodes as not visible, wanted to state here (at the root) that SLEAP will filter out all empty instances (with all non visible points). Labeling an instance as entirely occluded will not do anything (other than use-up your time): https://github.com/talmolab/sleap/blob/833c2d5bdcf4cfef4e0adc7569b8e2245494a8fa/sleap/nn/data/providers.py#L37-L59

[olinesn]

Since we seem to have gotten a lot of cases where people are marking all nodes as not visible, wanted to state here (at the root) that SLEAP will filter out all empty instances (with all non visible points). Labeling an instance as entirely occluded will not do anything (other than use-up your time):

https://github.com/talmolab/sleap/blob/833c2d5bdcf4cfef4e0adc7569b8e2245494a8fa/sleap/nn/data/providers.py#L37-L59

Thanks for the heads up!

[roomrys]

1070 implemented this for top-down models, but this still needs to be applied for other models.