[Detections] - enable `from_mmdetection` to extract `tracker_id` from MMDetection result

[rolson24]

Search before asking

• [X] I have searched the Supervision issues and found no similar feature requests.

Description

This is somewhat similar to #1308 in that it would be modifying from_mmdetection() to support TrackDataSamples. This would allow for easy integration with the MMTracking library, which has several trackers implemented in it, as well as integration with the Match and Segment Anything (MASA) model.

Use case

No response

Additional

I imagine it could be pretty simple and would only require checking if the input object has the pred_track_instances attribute, and then it could simply add the tracks with their track ID's into a Detections object.

Are you willing to submit a PR?

• [X] Yes I'd like to help by submitting a PR!

[SkalskiP]

Hi @rolson24 I just managed to plug in your MASA code into my football demo. I'd never be able to do it without your help. As for MASA: not bad, but I was hoping for a slightly better result. Would that allow us to plug in all MMTracking models?

https://github.com/user-attachments/assets/2c3bd164-82a4-40db-b12d-04442f534da0

[rolson24]

I don't think so. It seems like MASA is built on the trackers in the MMDet library, which are set up slightly different than the trackers in MMTracking. The trackers in MMTracking seem to output dicts of numpy arrays, while the trackers in MMDet output objects called TrackDataSample. Also I tried to test MMTracking and it seems that it does not support the newer versions of their own libraries. (Max MMCV version is 2.1.0 and newest is 2.2.0, and max MMDet version is 3.0.0 and newest is 3.3.0) So in order to use MMTracking with any recent version of CUDA and pytorch, a user would have to compile their own binaries. MMTracking has not had a commit since last April, so it seems like OpenMMLab doesn't have the bandwidth to support it right now.

So maybe not worth trying to support it if its a dead library anyway.

[rolson24]

As for the MASA performance, I also noticed it was pretty bad at handling occlusion and creating new track IDs. I looked a bit into the code and it seems like the actual tracking is a very simple tracking algorithm with no motion modeling. I think MASA has some pretty good potential if motion modeling was added, and a more strict matching pipeline was incorporated. I think the real benefit of MASA is its ability to plug into foundational open world detectors, and doesn't hold up as well in specific applications. I might try to play around and see if I can integrate it with a better tracker like BoTSORT with MASA as the ReID model.

[rolson24]

Maybe we should wait on the MASA integration because the library isn't quite there yet, and the tracking output will probably change.

If we wanted to integrate a strong tracking library that is still being maintained, I think BoxMOT could be a pretty good option. It supports some pretty good trackers and several ReID models and has a clean install process. I am also in the process of adding some new SOTA trackers to it.

[SkalskiP]

@rolson24 If we disregard MASA for the moment, do other trackers in MMTrack have a consistent output format?

[rolson24]

Yes. They output a dict containing track_results and det_results which are each dicts containing bbox_results and mask_results which in turn are lists of 2d numpy arrays where each array corresponds to a class. Track_ids are in bbox_results[0] and bboxes are in bbox_results[1].

The main issue with MMTracking is that it hasn't been updated since 2022, and so none of the dependancies are updated to work with any of the newer libraries.

It seems like all of the trackers from MMTracking have now been merged into MMDetection, which uses the same output format as MASA, and MMDetection has least had a release this year, so the dependancies are much more recent.

[SkalskiP]

@rolson24 in that case, could we expand from_mmdetection to consume tracking results as well?

[rolson24]

Ya it would be pretty easy. I can work on it right now.

[SkalskiP]

@rolson24 that's awesome! thanks a lot! 🙏🏻

[rolson24]

Ok I made the change and have been testing it, but the trackers and object detectors in MMDetection are not perfectly set up and the pre-trained weights seem to perform very badly on my test video.

from faster-RCNN and DeepSORT pretrained:

I can open the PR, but I the experience of using the trackers is less than ideal right now.

[SkalskiP]

Hm... I'm becoming increasingly convinced that tracking is the domain we should explore. If a library like MMDetection does not have trackers that work well out of the box, we could do something useful here.

[mikel-brostrom]

I am open the lend a hand here if you plan to integrate BoxMOT into Supervision @SkalskiP . Reach out with any questions you may have.