search

cguindel / eval_kitti

Tools to evaluate object detection results using the KITTI dataset.
57 stars 23 forks source link

Would it work for detecting only a subset of the validation split? #15

Open SM1991CODES opened 3 years ago

SM1991CODES commented 3 years ago

Hi, I was trying to use you eval script. But my detections are missing some frames, like if label_2 has a file 000000.txt, my detections need not have it (in case I did not detect anything, I don't save it). In that case, I think the code breaks as it tries to read this non-existent file from the results directory.

Is that the case?

Best Regards Sambit

cguindel commented 3 years ago

That is indeed correct: every frame in the validation set is expected to have a .txt result file, even if it is empty, as a way to avoid unwanted omissions.

It should be relatively easy to modify the script to skip non-available files if that is what you want. Just remove the lines where the gt_success and det_success variables are checked.

However, it is extremely unusual to have zero outputs in an image. Note that, for proper evaluation, the whole set of raw detections must be provided, including those with very low scores. Because of how the evaluation works, these instances should not hurt the final statistics even if they are highly unreliable. I recommend you to check if your method has any filter that is dismissing detections below a certain threshold and remove it.