reset_tracker while keeping existing object ids and features

[tornadoslims]

Thank you for creating such a powerful repository of libraries and utilities! I have had great success with it - it is pretty amazing how many different libraries you have enabled simple access to.

But I am having one issue and wonder if someone can point me in the right direction.

I am attempting to run an object detector every X frames, run one of the pytracking trackers the next Y frames, and then update the tracker from the object detector on frame X+Y without losing the existing object identifiers and without having to reinitialize the tracker.

An example would be like this:

Frame 1: Detector detects 10 objects, initializes tracker with bboxes from the 10 objects Frame 2-30: Tracker tracks the 10 objects Frame 31: Detector fires again, detects 8 objects, uses feature extractors to associate some detected objects to existing tracked objects, and other detected objects to new objects that need to start being tracked. Frame 31-60: Tracker tracks...

etc....

I have the pipeline working, but I am unable to find an entry point to "hint" the tracker that object 100 has a new gt bbox/centroid.

I think the ability to integrate your library into an existing object detector is what will make it so powerful. I might be missing something, and am just hoping someone can point me in the right direction. I've been googling and searching for the best way to do this for a week now - any help - however small, will be much appreciated! And I think will help the overall community as well.

Thank you.

[goutamgmb]

Hi.

In general, all of our trackers internally store the current state of the object being tracker in the variables 'pos', 'target_scale', and 'target_sz'. So if you want to manually change the state of the object, you could write a function which will overwrite these variables,

e.g.

def update_target_state(self, new_box):
    self.pos = torch.Tensor([new_box[1] + (new_box[3] - 1) / 2, new_box[0] + (new_box[2] - 1) / 2])
    self.target_sz = torch.Tensor([new_box[3], new_box[2]])
    self.target_scale = torch.sqrt(self.target_sz.prod() / self.base_target_sz.prod())

I agree that it is a useful functionality to have and will hopefully include it in the next release. However, note that this function will only reset the target position and size. In case the tracker had lost the target, and hence as a result the internal memory of the tracker was corrupted, it is possible that the tracker might loss the target again. In such cases, it would be beneficial to reset the memory of the tracker and update the model using the new clean data.

Regards Goutam

[tornadoslims]

Thank you for being so responsive and supportive. You have built an incredible library and I really appreciate you spending your time helping with questions about it.

I am going to close this issue - I've implemented @goutamgmb's answer and it works great. This code is really helpful and has now sent me down multiple exciting paths.

The below notes are just for people who are interested/working on integrating detectors like I am.

---

As expected, if the tracker has lost the object, updating the gt bbox and size at infrequent intervals, from an external call, while keeping the internal tracker memory the same is not the best way to go - you will rarely re-identify and object even if it hasn't left the frame.

But it is an excellent way to go if you are looking to have an intermediate step on a series of frequent intervals of frames, where you know objects will be moving in discrete directions (cars driving down a one way street). For anyone else working through something similar, also look at the max scores and flags as potential event signals.

I am currently working on testing:

1. If it's best to keep an internal map of {internal_ground_truth_ids->tracker_ids}, to enable re-initialization of all the trackers without losing an internal object id.

2. Update frame when flag 1 is normal and max score > x, but fire the detector and either re-initialize everything, or call the update_target_state() @goutamgmbwrote above if flag 1 != normal or max score < x

3. Attempting to value the tracker as significant of an authoritative record of truth as the detector. instead of having the detector run X frames and overwrite/blindly update the tracker objects, it seems more useful to have an algorithm that doesn't always treat the detector as ground truth. In my testing, letting the tracker run is often more accurate than allowing the detector to force the tracker into new positions every x frame. TLDR; these are some awesomely accurate trackers you've put together and made easy to use. Thanks again.

-TS

[goutamgmb]

Thanks for the nice words :) And thanks also for sharing your insights!

[hphnngcquan]

Thank you for being so responsive and supportive. You have built an incredible library and I really appreciate you spending your time helping with questions about it.

I am going to close this issue - I've implemented @goutamgmb's answer and it works great. This code is really helpful and has now sent me down multiple exciting paths.

The below notes are just for people who are interested/working on integrating detectors like I am.

---

As expected, if the tracker has lost the object, updating the gt bbox and size at infrequent intervals, from an external call, while keeping the internal tracker memory the same is not the best way to go - you will rarely re-identify and object even if it hasn't left the frame.

But it is an excellent way to go if you are looking to have an intermediate step on a series of frequent intervals of frames, where you know objects will be moving in discrete directions (cars driving down a one way street). For anyone else working through something similar, also look at the max scores and flags as potential event signals.

I am currently working on testing:

1. If it's best to keep an internal map of {internal_ground_truth_ids->tracker_ids}, to enable re-initialization of all the trackers without losing an internal object id.

2. Update frame when flag 1 is normal and max score > x, but fire the detector and either re-initialize everything, or call the update_target_state() @goutamgmbwrote above if flag 1 != normal or max score < x

3. Attempting to value the tracker as significant of an authoritative record of truth as the detector. instead of having the detector run X frames and overwrite/blindly update the tracker objects, it seems more useful to have an algorithm that doesn't always treat the detector as ground truth. In my testing, letting the tracker run is often more accurate than allowing the detector to force the tracker into new positions every x frame. TLDR; these are some awesomely accurate trackers you've put together and made easy to use. Thanks again.

-TS

Hi @tornadoslims, I am also interested in integrating a detector with a pytracking framework. Have you somehow run into a problem where the bounding box completely change the target object (for example when multiple cars approaching) I have integrated yolov7 and deepsort tracking strategy to DiMP and ran into this problem, quite annoying 😅