search

aimotive / mm_training

Multimodal model training on aiMotive Dataset
https://openreview.net/forum?id=LW3bRLlY-SA
MIT License
14 stars 4 forks source link

Evaluation - How to obtain Distance based metric for different scenes in the AiMotive val catalogue? #8

Open PaulSudarshan opened 3 months ago

PaulSudarshan commented 3 months ago

Please mention on how to obtain distance based metric for different scenes. @TamasMatuszka

TamasMatuszka commented 3 months ago

Hi @PaulSudarshan,

You can modify the x_range argument of this method to the distance you want to use.

PaulSudarshan commented 3 months ago

@TamasMatuszka thanks, and I suppose x_range value should be given in meters? Also can you provide some resources to know about the metrics used for evaluating the models in the attached image. metrics

TamasMatuszka commented 3 months ago

@PaulSudarshan Yes, the x_range value is given in meters. You can find some references to the used metrics here:

PaulSudarshan commented 3 months ago

@PaulSudarshan Yes, the x_range value is given in meters. You can find some references to the used metrics here:

Thanks for providing the resources. Can you clarify few more things : 1) test_num_preds - does it mean total predicted bounding boxes on the test set? 2) test_precision_2d , what does "2d" mean here? 3) test_precision_op, what does "op" mean here? 4) what does "test_cls_accuracy_bev" mean?

TamasMatuszka commented 3 months ago

@PaulSudarshan Sure.

  1. Yes, it is the number of predicted bounding boxes.
  2. 2D is a heritage of an old code, it used to be utilized for 2D metric calculation. However, the metrics are calculated in BEV space so the values returned by the evaluator are not 2D metrics in image space but rather the BEV metrics.
  3. 'op' determines the index of the optimal operating point. It maximizes precision*recall. Please refer to this method for concrete implementation.
  4. 'test_cls_accuracy_bev' refers to the classification accuracy.
PaulSudarshan commented 3 months ago

@PaulSudarshan Sure.

  1. Yes, it is the number of predicted bounding boxes.
  2. 2D is a heritage of an old code, it used to be utilized for 2D metric calculation. However, the metrics are calculated in BEV space so the values returned by the evaluator are not 2D metrics in image space but rather the BEV metrics.
  3. 'op' determines the index of the optimal operating point. It maximizes precision*recall. Please refer to this method for concrete implementation.
  4. 'test_cls_accuracy_bev' refers to the classification accuracy.

Thanks, so that means "test_precision_op" and "test_recall_op" determines the precision and recall for the most optimum value of score threshold given by "test_score_op", is that correct? @TamasMatuszka

If my understanding is correct, all these metrics are in BEV space right?

TamasMatuszka commented 3 months ago

@PaulSudarshan your understanding is correct in both cases.