search

KevinQian97 / ELECTRICITY-MTMC

ELECTRICITY: An Efficient Multi-camera Vehicle Tracking System for Intelligent City(CVPR 2020 AI City Challenge Track 3)
Other
113 stars 18 forks source link

Evaluate the model #5

Closed HanwenCao closed 4 years ago

HanwenCao commented 4 years ago

Hi. I have noticed that you have provided ways to test and train. Would you give me a hint on how to evaluate the whole system? I am very new to MTMC. I would like to compare the results with ground truth so that I can tell the potential causes of failure cases, and the situations where MTMC is hard. Thank you! Any suggestion is appreciated.

KevinQian97 commented 4 years ago

Hi Hanwen, IDP, IDR, and IDF1 are all common metrics for evaluation. This is the submission code for AI City, so we did not provide the evaluation code. We may update that part in the future. U can find the evaluation system on https://eval.aicitychallenge.org/aicity2020/submission. The illustration of these metrics is here: https://www.aicitychallenge.org/2020-data-and-evaluation/

HanwenCao commented 4 years ago

Thank you, Kevin. The links you provided are helpful. I have noticed that you have provided the code for visualization: here I was wondering if you have the usage of that? Thank you!!

HanwenCao commented 4 years ago

Hi, I ran test.sh on the test set and got "tracklets.txt" and "track3.txt". Is "tracklets.txt" the results of multi-object single-camera tracking? I was confused by looking at that results. Specifically, I found in the same frame of the same video(camera), there could be different vehicles that are assigned the same carID. For example, in frame 0 of camera042, different cars located at different areas in the image are assigned carID 1. Now I try to pick up some of these lines in "tracklets.txt" and list them below. ` 42 1 0 496 121 67 23 0 0 42 1 0 816 235 95 73 0 0 42 1 0 496 121 67 23 0 0 42 1 0 816 235 95 73 0 0 . . .

` I don't understand how they could be assigned the same carID by tracking algorithm, given the fact that they are in a single frame and located separately in image space. Also, I found many duplicated lines in "tracklets.txt" as shown above. Would you know the reason? Thank you!

KevinQian97 commented 4 years ago

Hi, hanwen. I guess that's because we are not sure if the specific vehicle only appears in a single view once. But maybe we should add such a filter (the same vehicle at the same time) to remove these FA samples.

KevinQian97 commented 4 years ago

Now close the issue

HanwenCao commented 4 years ago

Thank you!

chenjx1005 commented 3 years ago

Hi, hanwen. I guess that's because we are not sure if the specific vehicle only appears in a single view once. But maybe we should add such a filter (the same vehicle at the same time) to remove these FA samples.

tracklets.txt has 424146 lines. But after execute python ./identifier/preprocess/extract_img.py there are only 124661 images cropped. Is such many duplicated vehicles in a same frame of a same video a normal case?