Unsure whether this makes a difference but this is definitely in terms of neural networks.
the metric of use in many computer vision competitions is called average precision
another measure called the IoU measure stands for intersection over union overlap measure
this is the intersection value in the numerator divided by the union
assumption is that there is a ground truth box in there as well
the ground truth box could be taken from the spreadsheet document? but that also comes with the assumption that the goal is to do exactly as good as a person evaluating
False Positive as characterized by IoU
True Positive as characterized by IoU
False Negatives
Precision
measures how accurate the predictions are or the percentage that are correct
= true positive/(true positive + false positive)
recall
measures how good the method finds the positives
= true positive/(true positive + False Negative)
Average Precision (AP)
is the area under the precision-recall curve
? not entirely sure how this is plotted?
mAP
is the value calculated for all the classes
? is a class just a different setup of the model?
They recommend creating a validation dataset
this is for tuning the various parameters
and a test set
this is to test the performance of the model
Outlined tasks for Validation set
use mAP to select best model
? they say stable and consistent is desirable, but is the curve what they are talking about?
also that there is a way to predict whether the model is done training after a certain period also provided the results of mAP
Outlined Tasks for Test set
if false negatives/positives don't matter use the f1 score
if fp is unacceptable use the model with higher precision
if fn is unacceptable use model with higher recall
once you decide on the model try different confidence thresholds
In essence this piece describes a few metrics that are useful to determine whether the system is performing well, but there are a number of terms that are still unclear to me
first article to explore by Venkatesh Wadawadagi
https://www.kdnuggets.com/2020/08/metrics-evaluate-deep-learning-object-detectors.html
Unsure whether this makes a difference but this is definitely in terms of neural networks.
the metric of use in many computer vision competitions is called
average precision
another measure called the
IoU
measure stands forintersection over union
overlap measureFalse Positive as characterized by IoU
True Positive as characterized by IoU
False Negatives
Precision measures how accurate the predictions are or the percentage that are correct
= true positive/(true positive + false positive)
recall measures how good the method finds the positives = true positive/(true positive + False Negative)
Average Precision (AP) is the area under the precision-recall curve ? not entirely sure how this is plotted?![image](https://user-images.githubusercontent.com/11687631/139772548-c4a0cf03-c748-4e7b-a551-76034a2025f9.png)
mAP is the value calculated for all the classes
They recommend creating a validation dataset
Outlined tasks for Validation set
Outlined Tasks for Test set
In essence this piece describes a few metrics that are useful to determine whether the system is performing well, but there are a number of terms that are still unclear to me
Metrics
Unknown Terms
Their references