Closed akretz closed 2 years ago
If you ignore DPM detection with negative confidence, a lot of false positives get removed and the Hungarian algorithm performs better there than the k-d tree algorithm. Also, an IoU threshold of 0.05 seems really low to me. That results in a lot of bad matches with very small overlap and a lot of valid detections falsely removed as double detections. If you also change that to 0.2, you get the following result for the Hungarian algorithm:
IDF1 IDP IDR Rcll Prcn GT MT PT ML FP FN IDs FM MOTA MOTP IDt IDa IDm
MOT17-04-SDP 68.9% 78.0% 61.7% 77.0% 97.4% 83 47 26 10 974 10915 138 321 74.7% 0.154 81 51 12
MOT17-04-DPM 37.7% 56.4% 28.3% 43.5% 86.7% 83 10 40 33 3172 26865 208 354 36.4% 0.219 66 143 7
MOT17-05-DPM 39.8% 63.6% 28.9% 38.9% 85.5% 133 12 53 68 455 4228 66 112 31.3% 0.249 51 31 16
MOT17-05-FRCNN 55.8% 72.1% 45.5% 56.7% 89.9% 133 30 58 45 439 2997 58 71 49.5% 0.172 52 30 24
MOT17-11-FRCNN 53.4% 66.7% 44.6% 62.0% 92.8% 75 20 32 23 456 3586 49 51 56.6% 0.097 20 34 6
MOT17-05-SDP 55.6% 63.9% 49.3% 67.5% 87.5% 133 41 64 28 666 2249 89 120 56.6% 0.169 79 42 35
MOT17-02-FRCNN 34.6% 54.6% 25.3% 36.6% 78.9% 62 7 27 28 1816 11778 103 145 26.3% 0.130 45 68 13
MOT17-09-FRCNN 50.0% 66.6% 40.1% 59.3% 98.6% 26 7 17 2 45 2167 31 39 57.9% 0.107 17 21 7
MOT17-10-FRCNN 41.0% 45.1% 37.6% 61.9% 74.4% 57 16 36 5 2737 4889 293 342 38.3% 0.168 160 139 17
MOT17-04-FRCNN 55.8% 73.9% 44.9% 56.6% 93.1% 83 20 40 23 1995 20662 77 97 52.2% 0.109 12 68 3
MOT17-13-DPM 26.7% 72.6% 16.4% 19.9% 88.3% 110 9 27 74 308 9322 34 83 17.0% 0.272 11 30 7
MOT17-13-SDP 57.6% 70.9% 48.6% 58.1% 84.8% 110 47 27 36 1210 4875 116 194 46.7% 0.216 60 38 24
MOT17-02-SDP 39.3% 49.0% 32.9% 51.3% 76.4% 62 12 38 12 2944 9045 210 365 34.3% 0.203 105 105 12
MOT17-13-FRCNN 46.4% 51.5% 42.3% 61.0% 74.3% 110 37 50 23 2455 4538 316 326 37.2% 0.179 189 136 26
MOT17-02-DPM 20.9% 56.4% 12.8% 19.9% 87.7% 62 5 13 44 520 14875 51 105 16.9% 0.251 18 35 2
MOT17-10-SDP 48.9% 52.9% 45.4% 75.0% 87.3% 57 32 21 4 1401 3212 211 322 62.4% 0.209 118 84 8
MOT17-09-DPM 39.1% 45.2% 34.4% 60.5% 79.5% 26 4 18 4 831 2105 77 130 43.4% 0.272 47 30 2
MOT17-09-SDP 51.4% 61.5% 44.2% 68.2% 94.9% 26 11 15 0 196 1692 58 91 63.5% 0.153 40 24 8
MOT17-10-DPM 32.8% 54.1% 23.5% 37.9% 87.1% 57 7 18 32 724 7972 81 138 31.6% 0.251 24 60 6
MOT17-11-DPM 46.8% 62.7% 37.3% 55.6% 93.3% 75 12 26 37 375 4193 44 64 51.1% 0.220 13 34 5
MOT17-11-SDP 60.2% 65.8% 55.4% 76.0% 90.2% 75 30 33 12 780 2263 62 104 67.1% 0.153 26 41 13
OVERALL 48.3% 63.5% 39.0% 54.2% 88.2% 1638 416 679 543 24499 154428 2372 3574 46.2% 0.173 1234 1244 253
@akretz wanted to do this for ages, thanks ! I'm super busy this end of week but next week I'll have a look
@akretz Thank you for the contributions. Better late then never I guess ;-)
Hello everyone,
currently, there's only one MOT17 sequence used for benchmarking purposes. I've added a bash script which downloads the labels of all MOT17 sequences, unpacks them in the correct directory, runs the tracker on all sequences and finally evaluates all results. Also, I've removed short trajectories consisting of less than 10 frames from the output, because a lot of detection boxes are false positives and this should remove most of them.
I've also added an alternative matching algorithm using the Hungarian algorithm (also called Munkres assignment algorithm) instead of k-d trees. That algorithm, I thought, should perform better because it finds the minimum cost matching while the k-d tree algorithm only approximates a good solution. Running the MOT17 benchmark, I've found that the k-d tree algorithm has the following results
while the Hungarian algorithm has the following results
Since the Hungarian algorithm finds optimal matchings, it matches tracked objects to detection boxes more often, which results in fewer false negatives (FN), but also more false positives (FP). Also, we have more mostly tracked (MT, trajectories which are tracked for at least 80% of frames) trajectories.
In general, you can see that the tracker using the Hungarian algorithm performs slightly worse when the detector has a lot of false positives (compare the results which use the DPM detector) and slightly better when the detector doesn't have that many false positives. I suspect the Hungarian algorithm should be slightly better when we use YOLO detections, as the number of false positives tends to be rather small.
I've added a parameter which can switch between the k-d tree algorithm and the Hungarian algorithm and left the k-d tree algorithm as default. In order to evaluate which algorithm is a better fit for OpenDataCam, I guess further evaluation needs to be done. Using YOLO detections instead of detections given by the MOT17 benchmark for evaluation purposes is probably a good idea. Since traffic scenarios seem to be a major use case of OpenDataCam, using the UA-DETRAC dataset for future evaluations might be a good idea.