search

evalclass / precrec

An R library for accurate and fast calculations of Precision-Recall and ROC curves
https://evalclass.github.io/precrec
GNU General Public License v3.0
45 stars 5 forks source link

explanation of cb calculation #3

Closed fkgruber closed 7 years ago

fkgruber commented 7 years ago

Hi I'm using precrec to plot ROC results after running k-fold crossvalidation for different models. How are confidence intervals calculated when autoplot uses the option show_cb? is this based on the standard error and assuming normality or what.

thanks Fred

takayasaito commented 7 years ago

Hi Fred,

The evalmod function refers to three arguments for the confidence bands calculation.

  1. x_bins: It is an integer value to specify the number of minimum bins on the x-axis. It is used to define common supporting points for all datasets. The default x_bins value is 1000, which gives the default common supporting points as (0, 0.001, 0.002, ..., 0.999, 1). Precrec calculates the average points with 95% CI of the y-values on all the common supporting points.

  2. cb_alpha: It is a numeric value with range [0, 1] to specify the alpha value for the CI calculation. Precrec assumes that the errors are normally distributed. The default value is 0.05 so that 95% CI is calculated for each supporting point.

  3. calc_avg: It is a Boolean value to specify whether average curves should be calculated or not. The value is automatically determined depending on the input dataset unless explicitly specified.

Both the plot and the autoplot functions refer to one argument for confidence interval bands.

  1. show_cb: It is a Boolean value to specify whether point-wise confidence bounds are drawn. The points of all average values with 95% CI are connected to form confidence bands.

I'll add this description to the introduction page.

Cheers, Takaya

fkgruber commented 7 years ago

thanks!