OTB Experiment report metrics

[amoskalev]

Hi, my question is about how OTB experiment results are calculated, i.e. success and precision scores.

In the OTB benchmark, areas under success and precision curves are used, while in your code I see that you take the mean to calculate succ_score. Same with prec_score, I don't understand why do you calculate them like that.

Here is how metrics are calculated in ExperimentOTB:

succ_curve = np.mean(succ_curve, axis=0)
prec_curve = np.mean(prec_curve, axis=0)
succ_score = np.mean(succ_curve)
prec_score = prec_curve[20]

Shouldn't you take AUC instead of the mean value?

[huanglianghua]

AUC (Area-Under-Curve) is calculated by separating the curve into several discrete bins (nbins=50 in the code), taking each bin as a rectangle, and sum over the areas of these bins. In formular:

AUC = sum_i^nbins f(xi) * d_xi

where d_xi = 1 / nbins. So AUC is the mean over f(x_i)s.

The precision score is defined as the percentage of successfully tracked frames under the criterion: center error <= 20 (see OTB benchmark paper for details), which is prec_curve[20].