Evaluation metrics - Object detection

csudre commented 3 years ago

Is your feature request related to a problem? Please describe. Need for the following metrics when dealing with object detection problems

Intersection over Union (IoU) measurements can either be made at the level of the bounding boxes of the objects or of the segmentation themselves.

At the element level

Dice per element
Centroid distance

At the level of the image considering independent elements:

F1 score with IoU thresholds ranging from 0.5 to 0.95
Average precision with IoU thresholds ranging from 0.5 to 0.95 https://cocodataset.org/#detection-eval
Average recall with IoU thresholds ranging from 0.5 to 0.95 https://cocodataset.org/#detection-eval
Correlation of detected volumes
Number difference
F1 with minimum overlap of 1 voxel
Outline error https://link.springer.com/article/10.1186/1471-2342-12-17
Detection error https://link.springer.com/article/10.1186/1471-2342-12-17

Additional context This list of metrics is the output of the initial brainstorming of the metrics task force

mibaumgartner commented 3 years ago

Amazing list of detection metrics, thanks for all the effort of the metrics task force to collect them!

I am curious about the F1 with minimum overlap of 1 voxel metric, could someone add some papers which use that metric? (I would like to read them ^^)

Also a small addition: The Free-Response Receiver Operating Characteristic (FROC) might also be a metric which should be added (used by LUNA , DeepLesion and RibFrac)

iliathesmirnov commented 3 years ago

Hello everyone,

I am new here and would like to contribute to implementing some (or all) of the evaluation metrics on @csudre's list. Did anyone start working on them? If not, should I create a pull request (labelled [WIP]) and get going?

Thank you, Ilia

wyli commented 3 years ago

Hello everyone,

I am new here and would like to contribute to implementing some (or all) of the evaluation metrics on @csudre's list. Did anyone start working on them? If not, should I create a pull request (labelled [WIP]) and get going?

Thank you, Ilia

welcome and please go ahead! (I don't think anyone is currently working on them)

wyli commented 3 years ago

today we discussed the requirement of FROC curves for lesion based evaluation

Can-Zhao commented 1 year ago

Please check https://github.com/Project-MONAI/MONAI/blob/dev/monai/apps/detection/metrics/coco.py

Project-MONAI / MONAI

Evaluation metrics - Object detection #1132