search

cguindel / eval_kitti

Tools to evaluate object detection results using the KITTI dataset.
57 stars 23 forks source link

Does the alpha value matter for this? #16

Open AbdullahRJafar opened 3 years ago

AbdullahRJafar commented 3 years ago

Hey, I'm not sure how to get the alpha value of predictions so I just set it to 0, will this effect the evaluation?

Also I get a pretty good AP even though I thought I had a bunch of false detections. Although many of those false detections had a low confidence score. I think it doesn't count false detections when they confidence is low but I am not sure, hoping you can shine some light on that for me.

Lastly, I get something like 20% higher AP on this kitti eval, than I do on https://github.com/asharakeh/kitti_native_evaluation this one. They both look just like the one from the kitti website so I'm not sure what the difference would be. idk if you got any ideas about the last thing I mentioned.

cguindel commented 3 years ago

Hi @AbdullahRJafar,

  1. The alpha value can be computed from the yaw (rotated_y) angle and the 3D location of the obstacle. Anyway, it is only used to obtain the AOS, i.e., orientation evaluation; you can skip it by setting the value to -10 for all the detections (-10 is used to represent undefined in this case).
  2. Because of how the AP is defined, false positives with scores lower than the true positives should not affect the result. For a better understanding, you can review the AP metric thoughtfully, but, in short, I would say that you don't need to worry about them.
  3. At a glance, I don't see any significant differences between this code and the code in the repository you cite. Actually, the computeStatistics function seems functionally identical (which is expected since they both derive from the KITTI devkit). Can you please check if you are using the same test set in both cases? It seems like their code evaluates all the images found in result_dir, whereas this one expects a txt file containing the particular set of images used for evaluation. Maybe they are not identical, which will explain the difference.