Questions about kitti evaluation code

[tdzdog]

Recently, I studied the Kitti evaluation code in _kitti_object_evalpython. But I find a confusing thing. When handling the DontCare dt, there is a judgment of "metric == 0" a shown in the codes (line 249 in kitti_object_eval_python/eval.py)

if compute_fp:
        for i in range(det_size):
            if (not (assigned_detection[i] or ignored_det[i] == -1
                     or ignored_det[i] == 1 or ignored_threshold[i])):
                fp += 1
        nstuff = 0
        if metric == 0: ### here
            overlaps_dt_dc = image_box_overlap(dt_bboxes, dc_bboxes, 0)
            for i in range(dc_bboxes.shape[0]):
                for j in range(det_size):
                    if (assigned_detection[j]):
                        continue
                    if (ignored_det[j] == -1 or ignored_det[j] == 1):
                        continue
                    if (ignored_threshold[j]):
                        continue
                    if overlaps_dt_dc[j, i] > min_overlap:
                        assigned_detection[j] = True
                        nstuff += 1
        fp -= nstuff

This leads to that the fp predictions in DontCare are are ignored only for 2D bbox detection. Hence there are more fp for 3D object detection. Is this a bug or I missed something?

[MartinHahner]

https://github.com/open-mmlab/OpenPCDet/blob/26a16123cf820e29ed565f4ad53964c66f3ea030/pcdet/datasets/kitti/kitti_object_eval_python/eval.py#L249

[srimannarayanabaratam]

Well, the official kitti's evaluate_object.cpp [download their devkit_object] doesn't have this if metric == 0: check. They rather use the overlap corresponding to the metric: bev_box_overlap and d3_box_overlap (which is passed through a function pointer). So, indeed the code needs a fixing. It should not be a difficult task correcting it though.

A very simple fix can be something like this:

[tdzdog]

Well, the official kitti's evaluate_object.cpp [download their devkit_object] doesn't have this if metric == 0: check. They rather use the overlap corresponding to the metric: bev_box_overlap and d3_box_overlap (which is passed through a function pointer). So, indeed the code needs a fixing. It should not be a difficult task correcting it though.

A very simple fix can be something like this:

Thanks very much！

[github-actions[bot]]

This issue is stale because it has been open for 30 days with no activity.

[github-actions[bot]]

This issue was closed because it has been inactive for 14 days since being marked as stale.