search

xingyizhou / CenterTrack

Simultaneous object detection and tracking using center points.
MIT License
2.39k stars 526 forks source link

Questions about MOT results #119

Open liuqk3 opened 4 years ago

liuqk3 commented 4 years ago

Firstly, thanks for your grateful works! Here are my some questions about the results of MOT:

1) I tested on mot17test with provided models and commands, the results of 21 sequences contain 441750 boxes and 11208 tracklets. Also, I have downloaded the results of CTTrack17 from MOTChallenge. However, there are 429597 boxes and 3216 tracklets. Here comes the question: why the number of tracklets are so different? 2) Also, I tested on mot17test with provided models and commands based on public detections, there are 421943 boxes and 8445 tracklets in the results of 21 sequences. The only difference with private and public detections are the initialization of new tracklets. From the codes in src/lib/utils/tracker.py, line 83-101, we can see that the private detections that far away from any provided public detections will be filtered out. So I think the main difference between private and public rersults is FP and MOTA should be much close. However, the results in your paper , Table 1, shows that the public results posses 7511 lower FP but 38210 higher FN than private results. And the MOTA of public results is much lower too.

Can you please give some explanation to the above questions?

xingyizhou commented 4 years ago

Hi Qiankun, Thanks for the great questions.

  1. Please note that the submission of CTTrack17 has a slightly higher MOTA/ IDF1 results than our paper. This is because we have enabled a new feature of the framework: track rebirth. This is not contained in the current code. I should have published this, but just haven't found time to synchronize the code. I will try to release the updated version within two weeks.

  2. Sorry I did not fully understand your question. Feel free to ask in Chinese. Let me try to respond. Here is how I interpret the results: first of all, it is easy to understand that private detection can have much fewer FNs than public detection (because we can detect more objects than the provided detector can not). This is how the much higher MOTA comes from. Also, as our detector is more aggressive, it may contain more false positives, and the public detection model is able to filter out these false positives, resulting a lower FP. In general, the MOTA between public and private detection should be very different.

    Please let me know if you have further questions.

liuqk3 commented 4 years ago

Hi Xingyi, Thanks for your reply. Let me try to explain queation 2 in another way.

I haved done some works based on CenterTrack. With private detection, I got a desirable MOTA in MOT17. With public detection, I got a 0.8 lower MOTA. The results are almost the same with private and public detections. When tested with public detections, we can see from the line 83-101 in src/lib/utils/tracker.py that the detections are initialized as a tracklet only if they are close to provided public detections. However, the provided public detections are not filtered based on their confidence. So many public detections are enough to keep almost all private detections. The results in your paper, Table 1, shows that the MOTA of public detections are 5.9% lower than that of private results.

That's what puzzles me a lot.

agirbau commented 4 years ago

I'm just having the same issue, it seems that the public detections are only used for initialisation (in the current version of the code)?