Enable Partial AUC (and other pROC::auc() options) in roc_auc()

[gadenbuie]

I’m trying to calculate partial AUC using roc_auc(), but options passed to pROC::roc() intended for pROC::auc() aren’t making it through. My best guess is that this is related to the separate calls to pROC::roc() and pROC::auc() in roc_auc_binary() https://github.com/tidymodels/yardstick/blob/69608238ad2232f4088d3f6ae00fa5fa659e8fe0/R/prob-roc_auc.R#L153-L155

Here’s a small reprex using the “two_class_example”.

library(tidyverse)
library(yardstick)

data("two_class_example", package = "yardstick")

head(two_class_example)
#>    truth      Class1       Class2 predicted
#> 1 Class2 0.003589243 0.9964107574    Class2
#> 2 Class1 0.678621054 0.3213789460    Class1
#> 3 Class2 0.110893522 0.8891064779    Class2
#> 4 Class1 0.735161703 0.2648382969    Class1
#> 5 Class2 0.016239960 0.9837600397    Class2
#> 6 Class1 0.999275071 0.0007249286    Class1

proc_roc <- pROC::roc(two_class_example$Class2, response = two_class_example$truth)

pROC::plot.roc(proc_roc)

pROC AUC

pROC::auc(proc_roc)
#> Area under the curve: 0.9393

pROC Partial AUC

pROC::auc(proc_roc, partial.auc = c(1, 0.75))
#> Partial area under the curve (specificity 1-0.75): 0.1959

yardstick::roc_auc()

roc_auc(two_class_example, truth, Class2)
#> [1] 0.9393139

yardstick::roc_auc() Partial AUC

roc_auc(two_class_example, truth, Class2, options = list(partial.auc = c(1, 0.75)))
#> [1] 0.9393139

^{Created on 2019-03-21 by the reprex package (v0.2.1)}

Session info

``` r devtools::session_info() #> ─ Session info ────────────────────────────────────────────────────────── #> setting value #> version R version 3.5.2 (2018-12-20) #> os macOS High Sierra 10.13.6 #> system x86_64, darwin15.6.0 #> ui X11 #> language (EN) #> collate en_US.UTF-8 #> ctype en_US.UTF-8 #> tz America/New_York #> date 2019-03-21 #> #> ─ Packages ────────────────────────────────────────────────────────────── #> package * version date lib source #> assertthat 0.2.0 2017-04-11 [1] CRAN (R 3.5.0) #> backports 1.1.3 2018-12-14 [1] CRAN (R 3.5.0) #> broom * 0.5.0 2018-07-17 [1] CRAN (R 3.5.0) #> callr 3.2.0 2019-03-15 [1] standard (@3.2.0) #> cellranger 1.1.0 2016-07-27 [1] CRAN (R 3.5.0) #> cli 1.0.1 2018-09-25 [1] CRAN (R 3.5.0) #> colorspace 1.4-0 2018-10-15 [1] R-Forge (R 3.5.1) #> crayon 1.3.4 2017-09-16 [1] CRAN (R 3.5.0) #> curl 3.3 2019-01-10 [1] CRAN (R 3.5.2) #> desc 1.2.0 2018-05-01 [1] CRAN (R 3.5.0) #> devtools 2.0.1 2018-10-26 [1] standard (@2.0.1) #> digest 0.6.18 2018-10-10 [1] CRAN (R 3.5.0) #> dplyr * 0.8.0.9002 2019-02-21 [1] Github (tidyverse/dplyr@b23f50b) #> evaluate 0.13 2019-02-12 [1] CRAN (R 3.5.2) #> forcats * 0.3.0 2018-02-19 [1] CRAN (R 3.5.0) #> fs 1.2.6 2018-08-23 [1] CRAN (R 3.5.0) #> ggplot2 * 3.1.0 2018-10-25 [1] CRAN (R 3.5.0) #> glue 1.3.0.9000 2019-01-31 [1] Github (tidyverse/glue@8188cea) #> gtable 0.2.0 2016-02-26 [1] CRAN (R 3.5.0) #> haven 2.1.0 2019-02-19 [1] CRAN (R 3.5.2) #> highr 0.7 2018-06-09 [1] CRAN (R 3.5.0) #> hms 0.4.2 2018-03-10 [1] CRAN (R 3.5.0) #> htmltools 0.3.6 2017-04-28 [1] CRAN (R 3.5.0) #> httr 1.4.0.9000 2019-01-31 [1] Github (hadley/httr@307c7df) #> jsonlite 1.6 2018-12-07 [1] CRAN (R 3.5.0) #> knitr 1.22 2019-03-08 [1] CRAN (R 3.5.2) #> lattice 0.20-38 2018-11-04 [1] CRAN (R 3.5.2) #> lazyeval 0.2.1 2017-10-29 [1] CRAN (R 3.5.0) #> lubridate 1.7.4 2018-04-11 [1] CRAN (R 3.5.0) #> magrittr 1.5 2014-11-22 [1] CRAN (R 3.5.0) #> memoise 1.1.0 2017-04-21 [1] CRAN (R 3.5.0) #> mime 0.6 2018-10-05 [1] CRAN (R 3.5.0) #> MLmetrics 1.1.1 2016-05-13 [1] CRAN (R 3.5.0) #> modelr 0.1.2 2018-05-11 [1] CRAN (R 3.5.0) #> munsell 0.5.0 2018-06-12 [1] CRAN (R 3.5.0) #> nlme 3.1-137 2018-04-07 [1] CRAN (R 3.5.2) #> pillar 1.3.1 2018-12-15 [1] CRAN (R 3.5.0) #> pkgbuild 1.0.2 2018-10-16 [1] CRAN (R 3.5.0) #> pkgconfig 2.0.2 2018-08-16 [1] CRAN (R 3.5.0) #> pkgload 1.0.2 2018-10-29 [1] CRAN (R 3.5.0) #> plyr 1.8.4 2016-06-08 [1] CRAN (R 3.5.0) #> prettyunits 1.0.2 2015-07-13 [1] CRAN (R 3.5.0) #> pROC 1.12.1 2018-05-06 [1] CRAN (R 3.5.0) #> processx 3.3.0 2019-03-10 [1] CRAN (R 3.5.2) #> ps 1.3.0 2018-12-21 [1] CRAN (R 3.5.0) #> purrr * 0.3.1 2019-03-03 [1] standard (@0.3.1) #> R6 2.4.0 2019-02-14 [1] CRAN (R 3.5.2) #> Rcpp 1.0.0 2018-11-07 [1] CRAN (R 3.5.0) #> readr * 1.3.1 2018-12-21 [1] CRAN (R 3.5.0) #> readxl 1.1.0 2018-04-20 [1] CRAN (R 3.5.0) #> remotes 2.0.2 2018-10-30 [1] standard (@2.0.2) #> rlang 0.3.1 2019-01-08 [1] CRAN (R 3.5.2) #> rmarkdown 1.11 2018-12-08 [1] CRAN (R 3.5.0) #> rprojroot 1.3-2 2018-01-03 [1] CRAN (R 3.5.0) #> rvest 0.3.2 2016-06-17 [1] CRAN (R 3.5.0) #> scales 1.0.0 2018-08-09 [1] CRAN (R 3.5.0) #> sessioninfo 1.1.1 2018-11-05 [1] standard (@1.1.1) #> stringi 1.3.1 2019-02-13 [1] CRAN (R 3.5.2) #> stringr * 1.4.0 2019-02-10 [1] CRAN (R 3.5.2) #> testthat 2.0.1 2018-10-13 [1] CRAN (R 3.5.0) #> tibble * 2.0.1 2019-01-12 [1] CRAN (R 3.5.2) #> tidyr * 0.8.3 2019-03-01 [1] CRAN (R 3.5.2) #> tidyselect 0.2.5 2018-10-11 [1] CRAN (R 3.5.0) #> tidyverse * 1.2.1 2017-11-14 [1] CRAN (R 3.5.0) #> usethis 1.4.0.9000 2019-02-21 [1] Github (r-lib/usethis@0adfb5d) #> withr 2.1.2 2018-03-15 [1] CRAN (R 3.5.0) #> xfun 0.5.2 2019-03-14 [1] Github (yihui/xfun@d882a87) #> xml2 1.2.0 2018-01-24 [1] CRAN (R 3.5.0) #> yaml 2.2.0 2018-07-25 [1] CRAN (R 3.5.0) #> yardstick * 0.0.1 2017-11-12 [1] CRAN (R 3.5.0) #> #> [1] /Library/Frameworks/R.framework/Versions/3.5/Resources/library ```

[DavisVaughan]

It looks like in roc_auc(), rather than doing:

args <- quos(response = truth, predictor = estimate, levels = lvl)

curv <- eval_tidy(call2("roc", !!! args, !!! options, .ns = "pROC"))

res <- unname(pROC::auc(curv))

we could do:

args <- quos(response = truth, predictor = estimate, levels = lvl, auc = TRUE)

curv <- eval_tidy(call2("roc", !!! args, !!! options, .ns = "pROC"))

as.numeric(curv$auc)

Then the options would be passed through and we'd actually use that result rather than recomputing the auc separately.

This works because the partial auc and the auc are both stored in $auc and converting it to numeric is supported for both. We'd have to prevent the user from modifying the auc option like we do with the other 3. So in the docs add to "These options should not include response, predictor, or levels."

I think we are actually double computing the auc as it is, so that might improve the speed of the function as well.

[gadenbuie]

Awesome! Thanks for the fix and the speedy turnaround!

[github-actions[bot]]

This issue has been automatically locked. If you believe you have found a related problem, please file a new issue (with a reprex: https://reprex.tidyverse.org) and link to this issue.