search

bochinski / iou-tracker

Python implementation of the IOU Tracker
http://www.nue.tu-berlin.de
MIT License
689 stars 175 forks source link

Reproducing Results on DETRAC-Train Dataset with Mask-RCNN #6

Closed siyuanc2 closed 5 years ago

siyuanc2 commented 5 years ago

Hello,

I was trying to reproduce your results for the IOU-tracker on DETRAC-Train using detector Mask-RCNN. Although I have used the provided detection results and a new clone of your tracker, I keep getting results with MOTA of about 29.4 as opposed to the reported value of 34.36. Tracker iou-tracker-master + Detector frcnn by DETRAC-MOT Evaluation: PR-Rcll PR-Prcn PR-FAR|PR-MT PR-PT PR-ML|PR-FP PR-FN PR-IDs PR-FM| PR-MOTA PR-MOTP PR-MOTAL 34.8 41.0 0.36|28.8% 11.7% 26.7%|30139.2 187059.4 1006.6 995.2| 29.4 38.4 29.6 Would you mind looking at my attached logs below and see if I did anything wrong? Thank you very much for your help! log.txt

bochinski commented 5 years ago

Hi, I was finally able to reproduce your results. The reason is that apparently something goes wrong in the filtering process of the bounding boxes (dropping boxes from the ignored regions) in the detrac toolkit. If you run the evaluation on the raw detections provided in the download link you get indeed the results you posted. But the toolkit also reports a PR-MODA value of 62.96% for the detections, which is also way off.

BUT if you run the detector evaluation, you get a PR-MODA of 80.53%. The filtered detections are written back to the input files (MVI_*_Det_frcnn.txt...) If you run the tracking evaluation again you you get the right tracking results and a PR-MODA of 80.22%. (which seems close enough to the 80.53% of the detector evaluation, note that the threshold steps are a bit different).

So the toolkit seems to work a bit different as expected. You need to run the detector evaluation script at least once to get the proper preprocessing of the detections, then you can evaluate the tracker.

I wonder if thats the reason for the worse results on the test set.

siyuanc2 commented 5 years ago

This was indeed the issue... After running the detector evaluation I was able to reproduce the results! Thank you very much for pointing this out!

ngunauj commented 4 years ago

my train reproduce

Tracker iou-tracker + Detector frcnn by DETRAC-MOT Evaluation:
PR-Rcll PR-Prcn PR-FAR|PR-MT PR-PT PR-ML|PR-FP PR-FN PR-IDs PR-FM| PR-MOTA PR-MOTP PR-MOTAL 
 43.4  46.8  0.64|37.4% 11.4% 24.0%|53716.1 170699.7 1034.2 985.6|  33.9  45.4  34.1