Research metrics used for evaluating the performance of computer vision programs like this

[DevinBayly]

first article to explore by Venkatesh Wadawadagi

https://www.kdnuggets.com/2020/08/metrics-evaluate-deep-learning-object-detectors.html

Unsure whether this makes a difference but this is definitely in terms of neural networks.

the metric of use in many computer vision competitions is called average precision

another measure called the IoU measure stands for intersection over union overlap measure

• this is the intersection value in the numerator divided by the union
• assumption is that there is a ground truth box in there as well
• the ground truth box could be taken from the spreadsheet document? but that also comes with the assumption that the goal is to do exactly as good as a person evaluating

False Positive as characterized by IoU

True Positive as characterized by IoU

False Negatives

Precision measures how accurate the predictions are or the percentage that are correct

= true positive/(true positive + false positive)

recall measures how good the method finds the positives = true positive/(true positive + False Negative)

Average Precision (AP) is the area under the precision-recall curve ? not entirely sure how this is plotted?

mAP is the value calculated for all the classes

• ? is a class just a different setup of the model?

They recommend creating a validation dataset

• this is for tuning the various parameters and a test set
• this is to test the performance of the model

Outlined tasks for Validation set

• use mAP to select best model
  • ? they say stable and consistent is desirable, but is the curve what they are talking about?
  • also that there is a way to predict whether the model is done training after a certain period also provided the results of mAP

Outlined Tasks for Test set

• if false negatives/positives don't matter use the f1 score
• if fp is unacceptable use the model with higher precision
• if fn is unacceptable use model with higher recall
• once you decide on the model try different confidence thresholds

In essence this piece describes a few metrics that are useful to determine whether the system is performing well, but there are a number of terms that are still unclear to me

Metrics

• precision
• recall
• F1
• Average Precision AP
• mAP -IoU

Unknown Terms

• classes
• IoU Threshold

Their references

https://arxiv.org/pdf/1711.00164.pdf
https://www.researchgate.net/publication/343194514_A_Survey_on_Performance_Metrics_for_Object-Detection_Algorithms/link/5f1b5a5e45851515ef478268
https://github.com/ultralytics/yolov3/issues/898
https://www.pyimagesearch.com/2016/11/07/intersection-over-union-iou-for-object-detection/