Error evaluating my 3D only tracking algorithm.

[anti-destiny]

Hello. Thank you for your great work. I found a problem when evaluating my Multi-object Tracking algorithm with the KITTI dataset, and I think it may be a bug.

Since my detector only predicts 3D bounding boxes, the 2D bounding boxes are encoded as (-1, -1, -1, -1) in my prediction result files, which leads to 0 false positives when evaluating. An example of the evaluation report is given below.

Multiple Object Tracking Accuracy (MOTA)                                  0.8116
Multiple Object Tracking Precision (MOTP)                                 0.7274
Multiple Object Tracking Accuracy (MOTAL)                                 0.8121
Multiple Object Detection Accuracy (MODA)                                 0.8122
Multiple Object Detection Precision (MODP)                                3.5486

Recall                                                                    0.8366
Precision                                                                 1.0000
F1                                                                        0.9110
False Alarm Rate                                                          0.0000

Mostly Tracked                                                            0.6835
Partly Tracked                                                            0.2554
Mostly Lost                                                               0.0612

True Positives                                                             10548
Ignored True Positives                                                      1638
False Positives                                                                0 < here!!
False Negatives                                                             2060
Ignored False Negatives                                                     1168
ID-switches                                                                    7
Fragmentations                                                                56

Finally, I found the reason is that when the height of a predicted 2D bounding box is less than a specific threshold, it's ignored by the evaluation tool. And in my case, the height of bounding boxes is always zero. I think that it may be better to allow evaluating 3D only tracking algorithms since there are many methods can not predict 2D bounding boxes.

[xinshuoweng]

Yes, you are right! That is a good point. In fact, the evaluation code was extended from the official KITTI detection evaluation code so it does have the component of checking the height of the box. A simple workaround for this is to fill some dummy but reasonable numbers for the 2D box, e.g., [xmin, ymin, xmax, ymax] = [10, 10, 100, 100]. Or one can simply project your 3D box into 2D given the camera matrix, which should be feasible to do for any 3D tracking methods