search

openvinotoolkit / open_model_zoo

Pre-trained Deep Learning models and demos (high quality and extremely fast)
https://docs.openvino.ai/latest/model_zoo.html
Apache License 2.0
4.1k stars 1.37k forks source link

Bug causes degraded performance with latest version of multi_camera_multi_target_tracking_demo #3588

Open zvif opened 2 years ago

zvif commented 2 years ago

We have recently upgraded our OpenVino version from 2021.2 to 2022.2. We are running the Python demo of multi_camera_multi_target_tracking_demo installed using https://pypi.org/project/openvino-dev/ Ubuntu* 20.04 long-term support (LTS), using WSL Our command line: python multi_camera_multi_target_tracking_demo.py -i ./Data/filename.mp4 --config ./configs/person.py -m "./intel/person-detection-retail-0013/FP32/person-detection-retail-0013.xml" --m_reid "./intel/person-reidentification-retail-0286/FP32/person-reidentification-retail-0286.xml" We observe the following:

  1. What is most worrying is that the performance of the tracker is degraded - mostly by getting less tracks since many more tracks are merged. After investigating this issue - I found a problematic area but I haven't managed to figure it out. In the file mc_tracker in function _compute_detections_assignment_cost line 526 the cluster distance is calculated : reid_dist_clust = clusters_vec_distance(self.tracks[idx].f_clust, features[j]) I have found that this value (in the latest version) is sometimes 0.0. This I found is caused by a cluster being updated to the exact embedding of the current frame (therefore obtaining an exact 0.0 distance) but I can't figure out how the cluster can be updated in that same frame.
  2. If we run the command line we get the message: FATAL: exception not rethrown Aborted If we add "--history_file "./Data/filename_b.json" this message disappears.

This is really important to us since we created a solution based on this code base. Adding our test clip: https://user-images.githubusercontent.com/68502115/200169839-42426075-04b2-48b2-9e2e-f1bbdfb8725d.mp4 Thanks Zvi

zvif commented 2 years ago

Adding that this was compared to the version openvino 2021.4.2 running on the same computer. Attaching the results for the clip attached. filename_json_results.zip

Wovchena commented 2 years ago

I can’t set up the GUI for WSL to reproduce the problem because I have Windows Version 10.0.19044.2130, but 21364+ is required: https://github.com/microsoft/wslg/wiki/Diagnosing-%22cannot-open-display%22-type-issues-with-WSLg#verify-you-are-running-on-windows-build-21364. I tried running on bare Windows, but I couldn't reproduce your FATAL: exception not rethrown.

Adding that this was compared to the version openvino 2021.4.2

Are the results for 2021.4.2 the same as for 2021.2?

You can try finding the problem on your own. Try running the old demo version with new openvino. This will tell you if the problem is in demo or in openvino. Also try running different models: detector and reid models that correspond to 2021.2 and 2022.2. After that it will be possible to focus investigation on one of the components.

zvif commented 2 years ago

I compared the results of running OV 2021.4.2 with the results of running OV 2022.2 with the demo code of 2021.4.2. The results are nearly the same (very small detector differences even though the models are the same)! This means that the problem is with the new demo code that uses the new api (either bug in new engine or in the code that calls the new api). From what I understand the real benefit of using the 2022 version is the new api.

Wovchena commented 2 years ago

There aren't many of the commits to multi_camera_multi_target_tracking_demo since 2021.4.2: https://github.com/openvinotoolkit/open_model_zoo/commits/master/demos/multi_camera_multi_target_tracking_demo/python You can now freeze OV at 2022.2 and try different demos commits. That will show you which change broke the demo for you.

saurabhmj11 commented 1 year ago

It seems that the problem you are encountering is related to a bug in the multi_camera_multi_target_tracking_demo Python demo when running it with OpenVINO 2022.2. Specifically, the performance of the tracker is degraded due to some tracks being merged incorrectly, and this is caused by a cluster being updated to the exact embedding of the current frame, resulting in a distance of 0.0.

One possible solution to this issue is to use the "--history_file" option when running the demo, as you mentioned. This option saves the tracking history to a JSON file, which can be used to re-run the demo with the same parameters and reproduce the same results. This can help to identify the cause of the issue and potentially fix it.

Another option is to try using an earlier version of OpenVINO, such as 2021.2, which did not have this issue. You can also check if there is a newer version of the demo available that addresses this issue.

If you are unable to solve the issue using these methods, you may need to contact the OpenVINO support team for further assistance.