Change in motion of object and 1 missed det led to change in tracking ID.

[alaap001]

Search before asking

• [X] I have searched the Yolo Tracking issues and found no similar bug report.

Question

Hey @mikel-brostrom Thanks for the great repo an help so far.

I have a curious case related to association(if I am not wrong) I have been thinking about how to solve it, and any advice from an expert would be appreciated.

Brief of issue:

I have a human walking in 1 direction to load some goods on a truck and the ID is being properly tracked to 0.

Later he instantly chose to go in the opposite direction, and as he does that, the detector misses detection for 1 frame and 1 frame only. As soon as I get the new bbox (which is different in size and in different direction) the tracker assigns it a different id of 1.

Now what I cannot fathom is how just 1 missed det led to a change in id. I checked the boxes, and IOU is > 0.3 and human hasn't moved too far, only in the opposite dir. The size of bbox changes due to the angle so I figured crop could have different features that ReId failed to associate.

But I suspect it could be kalman filter but I tried setting w_association_emb=0.99 to put more weight on ReID in hope that ReID will surely work but that did not help either. Different REID models also not helping. [Clip to osnet x1]

I have tried playing with inertia, max-age, and other params with different values but the problem persists.

How could we have solved this problem if we were to? it will be a common thing that object changes its motion and in btw missed 1 or 2 dets, new det should still be assigned the same id but here in this case it is not.

I created a reference image to make it easier to understand:

Thank you.

[alaap001]

+1 I forgot to mention this is happening for DeepOCSort and HybridSort, but BotSort is working fine. Could it be the way ReID is being used in both?

[mikel-brostrom]

I would say it is probably more of a Kalman Filter configuration issue. What is your FPS on the camera?

[alaap001]

it is 15 FPS,

Is there any way I can test if it is Kalman Filter issue or not?

[mikel-brostrom]

You could try to scale the process noise covariance. In DeepCOSORT these lines would look like this:

        self.kf.Q[4:6, 4:6] *= self.Q_xy_scaling
        self.kf.Q[-1, -1] *= self.Q_s_scaling

This is because the predictions can be too strict and stiff for your specific use-case, especially if it is a dynamic environment where the bounding box sizes and positions change a lot.

[alaap001]

Ok so if I understood you correctly, Kalman filter has 2 noise factors, Q and R, where Q represents uncertainty in model prediction. And we are trying to scale Q because my model bbox can change cz of dynamic env where changes can be abrupt and sudden.

I tried to make changes but Q_xy_scaling and Q_s_scaling are not defined AttributeError: 'KalmanBoxTracker' object has no attribute 'Q_xy_scaling'

Can you help me understand How should these be defined?

for testing, I tried different values fro scaling from 0.01 to 0.05, 0.5 to even 1.0. All resulted in the same output

Thanks a lot for the help.

[mikel-brostrom]

I tried to make changes but Q_xy_scaling and Q_s_scaling are not defined AttributeError: 'KalmanBoxTracker' object has no attribute 'Q_xy_scaling'

Yup, this is not implemented so you would need to find your way. To start with, you could just define them as class variables and play around with them

[mikel-brostrom]

If you want to know more check out this:

https://github.com/mikel-brostrom/boxmot/pull/1484

[alaap001]

Thanks for the reference, Ya I've been playing around with a small range and let's see what comes up.

1 Question, isn't Kalman filter covarience working the same for both DeepOCSort and BOTSort? If it was Kalman, Why would it work with BotSort? (Altho in BotSort tracker loses track as soon as human goes out of frame even after increasing track_buffer to 600) But that's a sep issue.

[edblu1]

Is there any way I can test if it is Kalman Filter issue or not?

What helped me debug the behavior of the KF is to also return the KF bbox predictions from the update method and then also plot them on the video output, this way you would see what the Kalman Filter is doing during those phases of missed detections and if this is the issue :)

[dalerxli]

您好！您的邮件已收到，谢谢。

[alaap001]

Is there any way I can test if it is Kalman Filter issue or not?

What helped me debugging the behaviour of the KF is to also return the KF bbox predictions from the update method and then also plot them on the video output, this way you would see what the Kalman Filter is doing during those phases of missed detections and if this is the issue :)

This sounds like a good idea, I tried different covariance values, but unfortunately, that didn't help fix the problem. Let me try plotting bbox and see if that helps.

Will get back here in sometime, if anyone has already done something similar to debug these tracking issues, please tag in this issue, it'll be great to have useful resources.

Thanks for the help. :)

[mikel-brostrom]

What helped me debugging the behaviour of the KF is to also return the KF bbox predictions from the update method and then also plot them on the video output

Yup, this is a good recommendation

[alaap001]

Thanks a lot for the suggestion, I tried what you asked and plotted Kalman filter and changed values of Q matrix to see impact, even tried Q to be an identity matrix.

Here are the problems I noticed with KF:

Issue

In this blue box is of KF and box with id: 3 is old tracked id and id: 5 is FP switch of id.

Here as we can see, id: 3 was the original id being tracked, but as soon as a human walked off the van and we missed 1 det in between, the new det came in and got assigned a diff id: 5 instead of same. The KF of the original id: 3 is expecting him to move ahead but he moved back and a new id got assigned even though overlap is significant here.

Simlarly

In this blue box is of KF and box with id: 2 is old tracked id and id: 3 is FP switch of id.

Note in the above image id: 2 was the original track id. Later after 2,3 missed dets we got a new detection which is of same person but got assigned a different id. At this point KF of old det is way off the original path but still has some degree of overlap.

Is there anything we can do in tracking algo to fix issues like these in an environment where movement is mostly unpredictable??

Using identity matrix and changing scale factor for Q has helped in couple of cases but not in the above 2 where a new det (with missed dets in btw) get's assigned a new id for the same object.

PS: A good article to understand KF better: The Kalman Filter: An algorithm for making sense of fused sensor insight Thanks. :)

[mikel-brostrom]

Multiple predictions ahead could help here

[alaap001]

Multiple predictions ahead could help here

Hey, I'm sorry, I didn't get you. Do you mean a better detector?

[mikel-brostrom]

Not sure what value you set max_age to, but the number of predictions ahead made when the track is lost is exactly max_age. During this period, the tracker relies solely on its prediction model without any new observations to update its state.

https://github.com/mikel-brostrom/boxmot/blob/dccde97ddc37f051089767517915eeb721740cd3/boxmot/trackers/deepocsort/deep_ocsort.py#L163-L165

[alaap001]

Oh ok, here are my params:

tracker = DeepOCSORT(
    model_weights = Path('clip_market1501.pt'),  # which ReID model to use
    device = 'cuda:0',
    det_thresh = 0.3,
    min_hits = 3,
    w_association_emb=0.7,
    iou_threshold = 0.2,
    # embedding_off = False,
    fp16 = False,
    # max_obs = 90,
    inertia=0.4,
    max_age=60,
    # cmc_off = False,
    # aw_off = False,
    # alpha_fixed_emb = 0.95, 
    # per_class=False,
    # new_kf_off = False 
)

I had set it to 60 hoping that we would maintain the track even if the human is hidden for 3 to 4 seconds (15FPS video)

[mikel-brostrom]

I had set it to 60 hoping that we would maintain the track even if the human is hidden for 3 to 4 seconds (15FPS video)

That is right. It keeps it, but it is lost, although still making predictions for it... And lives as lost for 60 frames before removed

[mikel-brostrom]

The easiest solution for you IMO, would be to do a hyperparameter search. There are MANY parameters in these trackers and it is usually not trivial to figure out what is going wrong.

[alaap001]

Ya, I was thinking of using RayTune. Do you have any better suggestions based on your experience for this?

Also, do you recommend training of ReID model? I was honestly hoping ReID to recognize that this is the same object as shown in image 1 but somehow it didn't.

[mikel-brostrom]

You have an evolve.py script in this repo 🚀

[alaap001]

@mikel-brostrom

do you think we can use something like BPReid to make the tracking more robust? How hard it'll be to integrate this, and is it worth it to improve DeepOCSort performance?

[github-actions[bot]]

👋 Hello, this issue has been automatically marked as stale because it has not had recent activity. Please note it will be closed if no further activity occurs. Feel free to inform us of any other issues you discover or feature requests that come to mind in the future. Pull Requests (PRs) are also always welcomed!