Run Tracker Untill Lose The Object.

[RGuven]

First of all, thank you for this awesome repo. My problem is related to object detection. I'm using yolov7 as a detector.

I am able to do track in every n frame at the moment. There is no problem here. But I wonder how many frames I can track before lose the object. So my plan is to do track that object until lose it, if the tracker loses the object, run the detector again.

Can I get something like a match score or prediction score from the tracker to apply these my thoughts? I'm also open to other ideas. How can ı do that?

Thanks in advance.

[javiber]

Hi @RGuven glad you are liking Norfair!

I'm afraid there is no way of knowing when the tracker lost an object without having the detections. The tracker can estimate the movement indefinitely but without detection to compare it to, you can't know how good the estimate is.

However, if you are just interested in running an experiment to select a good n, then you can run the detector on each frame but only pass them to the tracker every n frames. That way the tracker is forced to estimate multiple frames but you have the detections to compare it against.

The pseudo-code would be something like this:

for i, frame in enumerate(video):
    detections = yolo(frame)
    if i % n == 0:
        tracked_objects = tracker.update(detections, period=n)
    else:
        tracked_objects = tracker.update(period=n)
    error = some_distance(detections, tracked_objects)
    if error > threshold:
        print(f"Object was lost at frame {i}. n={n}")

[RGuven]

Hi @javiber, Thank you so much for your quick and descriptive reply.

I should select an "n" variable in your pseudo-code. Your code will work like running the detector every nth frame and then starting to track till another modulo operator if it satisfies the condition.

What I actually want, controlling the tracker via prediction score from Kalman Filter or somewhere idk. The code in my mind is something like this:

for i, frame in enumerate(video):
    pred_score=from_Kalman_or_somewhere()

    if pred_score < threshold :
        detections = yolo(frame)
        tracked_objects = tracker.update(detections)
    else:
        # new question arises :) 
        # in here if I have to feed the tracker, what if I feed with last 
        # detection data ` (obj.last_detection.data)`  is located in the last inherited tracker obj.

         tracked_objects = tracker.update()

[javiber]

My pseudocode was meant to run an experiment to select an n that would work in your case and then use it, but if you need a variable n that's not going to work.

The problem I see is how you could calculate pred_score without comparing it against the detections from YOLO (which implies running YOLO on every frame which defeats the whole purpose of skipping frames).

Without researching about it, the closest thing that comes to my mind is "inverting" the logic, starting by running YOLO on every frame and track as usual but checking how close the Kalman filter was to the detection on each frame. Once the Kalman filter is "close enough" skip the detections on the next n frames. I can see a couple of issues with this approach though:

1. Because you need to decide in advance I'm going to skip n frames now, you are trusting that the object will move in a predictable trajectory (which means constant speed)
2. Actually calculating how close the Kalman filter was is not straightforward from outside the tracker. The tracker would need to provide that for you (maybe the sum of the distances of the matches can be used)

In summary:

• I don't know how to calculate the necessary pred_score. If you find some technique we are open to discussing it.
• You are welcome to try the idea from above but I'm afraid that it will not be too easy to implement while also having a low chance of working.

[RGuven]

Okay I see. Thank you for your thoughts and directions. I was just wondering if such an implementation is possible.

I'm closing the issue now. If I find something new in my research, I will reopen the issue again and share my informations.