Also, the official kitti evaluate_object.cpp [download their dev kit] does not "explicitly" filters the fps for metric=0. Rather, they use the box_overlap method corresponding to the respective metric (image2D/bev/3D).
So, I suggest a simple fix for the same as shown below.
if compute_fp:
# count fp
for i in range(det_size):
if (not (assigned_detection[i] or ignored_det[i] == -1
or ignored_det[i] == 1 or ignored_threshold[i])):
fp += 1
nstuff = 0
# do not consider detections falling under neutral zones as fp
if metric==0:
overlaps_dt_dc = image_box_overlap(dt_bboxes, dc_bboxes, 0)
elif metric==1:
overlaps_dt_dc = bev_box_overlap(dt_bboxes, dc_bboxes, 0)
else: # metric==2:
overlaps_dt_dc = d3_box_overlap(dt_bboxes, dc_bboxes, 0)
for i in range(dc_bboxes.shape[0]):
for j in range(det_size):
if (assigned_detection[j]):
continue
if (ignored_det[j] == -1 or ignored_det[j] == 1):
continue
if (ignored_threshold[j]):
continue
if overlaps_dt_dc[j, i] > min_overlap:
assigned_detection[j] = True
nstuff += 1
fp -= nstuff
Line https://github.com/traveller59/kitti-object-eval-python/blob/master/eval.py#L277 only applies the nstuff filter when metric =0 i.e; evaluating on image 2D bbbox. The current implementation gives poor performance when evaluation is done through bev or 3D iou.
Also, the official kitti evaluate_object.cpp [download their dev kit] does not "explicitly" filters the fps for metric=0. Rather, they use the box_overlap method corresponding to the respective metric (image2D/bev/3D).
So, I suggest a simple fix for the same as shown below.