Evaluation Metrics

[witignite]

This issue is about the evaluating the detection of a model. I do not completely understand the full code right now, but basically the code is right there and we may want to look into the code at train/kitti_eval/evaluate_object_3d_offline.cpp.

The followings are some of the excerpts from here.

Summary

• There are 2 types of evaluation: detection and localization
  • In 3D detection evaluation, we use 3D bounding box overlap to determine true/false positives/negatives.
  • In 3D localization evaluation, the bounding boxes (that our model predicts) are first projected to the bird’s eye view plane. Then IoU is evaluated on oriented 2D boxes.
• There are 3 level of difficulties defined, or performance metrics:
  • EASY
  • Min bounding box height: 40 Px
  • Max occlusion level: Fully visible
  • Max truncation: 15 %
  • MODERATE
  • Min bounding box height: 25 Px
  • Max occlusion level: Partly occluded
  • Max truncation: 30 %
  • HARD
  • Min. bounding box height: 25 Px
  • Max. occlusion level: Difficult to see
  • Max. truncation: 50 %

Results

• Pre-trained model

  car_detection AP: 96.482544 90.305161 87.626389
  pedestrian_detection AP: 84.555908 76.389679 72.438660
  cyclist_detection AP: 88.994469 72.703659 70.610924
  Finished 2D bounding box eval.
  
  car_detection_ground AP: 88.074974 84.163605 76.563362
  pedestrian_detection_ground AP: 71.315178 65.249374 58.500385
  cyclist_detection_ground AP: 79.210487 60.521530 56.146439
  Finished Birdeye eval.
  
  car_detection_3d AP: 83.864120 70.871918 63.609463
  pedestrian_detection_3d AP: 64.777824 57.965805 51.054234
  cyclist_detection_3d AP: 74.810394 55.703781 52.185024
  Finished 3D bounding box eval.

• Trained model (epoch ~50)

  car_detection AP: 96.482544 90.305161 87.626389
  pedestrian_detection AP: 84.555908 76.389679 72.438660
  cyclist_detection AP: 88.994469 72.703659 70.610924
  Finished 2D bounding box eval.
  
  car_detection_ground AP: 88.406845 83.801437 76.078979
  pedestrian_detection_ground AP: 72.838211 66.678329 59.833534
  cyclist_detection_ground AP: 80.104805 61.279713 57.375103
  Finished Birdeye eval.
  
  car_detection_3d AP: 82.356071 68.852745 62.274784
  pedestrian_detection_3d AP: 68.141731 59.950481 52.457260
  cyclist_detection_3d AP: 72.713387 55.370434 52.313919
  Finished 3D bounding box eval.