Documentation and cutpointr output suggestion

[sheilasaia]

Thank you for developing this really great package! I have three suggestions/enhancements for this package.

First, the output to the cutpointr( ) function contains a column that is named after the metric that the user chooses. So if metric = "youden" then the column will be called youden. I suggest changing this column name to metric_value and adding a new column called metric that basically gives the character string of the metric used (in this case youden). This will enable the users to run the analysis for mutiple metrics, bind results together by row, and use tidyverse functions to wrangle the output.

Second, when adding in subgroups the cutpointr( ) output has a new column subgroup and therefore this no longer matches the function output from the non-subgroup cutpointr( ) case. I suggest adding a subgroup column with the character string none or something similar to the non-subgroup case so that these two outputs can be bound if the user wants to do this.

Last, this is sort of a question and a suggestion. Can you please change the boot_stratify explanation to say something like "...keeping a proportionate number of positives and negatives as in the full dataset when resampling"? As it's written now, it sounds like the number of positives is exactly equal to the number of negatives.

Thanks again!

[Thie1e]

Hi, thanks for taking the time to write this detailed comment.

Regarding columns in the output, I have to admit that I'm quite sure that I'm not going to change that in the near future, just because it would break a lot of older code and I would have to rewrite many of the internal functions. The way the main metric is reported now is consistent with the other metrics that also have their own columns (I know, their names are not dynamic). Another reason to do it this way was to not make the output any wider, as it is already very wide.

What you seem to be concerned with is binding rows. I agree that using base R methods this might be tricky, but with bind_rows it works quite well in both cases (different optimized metrics and with/without subgroups). I have included some examples below. Maybe you've already done it that way anyway, I just wanted to illustrate how I would do it. Or is there some specific kind of wrangling the output that still can't be done?

You're right about the documentation of boot_stratify, thanks. Of course I meant so say 'keeping the proportion of positives and negatives constant"..,

library(tidyverse)
library(cutpointr)

# Bind rows of cutpointr objects with and without subgroup ----------------

cp1 <- cutpointr(suicide, dsi, suicide, metric = accuracy)
#> Assuming the positive class is yes
#> Assuming the positive class has higher x values
cp2 <- cutpointr(suicide, dsi, suicide, gender, metric = accuracy)
#> Assuming the positive class is yes
#> Assuming the positive class has higher x values
bind_rows(cp1, cp2)
#> # A tibble: 3 x 18
#>   direction optimal_cutpoint method          accuracy      acc sensitivity
#>   <chr>                <dbl> <chr>              <dbl>    <dbl>       <dbl>
#> 1 >=                       6 maximize_metric 0.951128 0.951128    0.444444
#> 2 >=                       6 maximize_metric 0.956633 0.956633    0.444444
#> 3 >=                       8 maximize_metric 0.957143 0.957143    0.333333
#>   specificity      AUC pos_class neg_class prevalence outcome predictor
#>         <dbl>    <dbl> <fct>     <fct>          <dbl> <chr>   <chr>    
#> 1    0.987903 0.923779 yes       no         0.0676692 suicide dsi      
#> 2    0.994521 0.944647 yes       no         0.0688776 suicide dsi      
#> 3    1        0.861747 yes       no         0.0642857 suicide dsi      
#>   data               roc_curve                 boot  subgroup grouping
#>   <list>             <list>                    <lgl> <chr>    <chr>   
#> 1 <tibble [532 x 2]> <roc_cutpointr [13 x 10]> NA    <NA>     <NA>    
#> 2 <tibble [392 x 2]> <roc_cutpointr [11 x 10]> NA    female   gender  
#> 3 <tibble [140 x 2]> <roc_cutpointr [11 x 10]> NA    male     gender

# Bind rows if different metrics were maximized and avoid NAs -------------

cp1 <- cutpointr(suicide, dsi, suicide, metric = youden) %>% 
    add_metric(list(accuracy))
#> Assuming the positive class is yes
#> Assuming the positive class has higher x values
cp2 <- cutpointr(suicide, dsi, suicide, metric = accuracy) %>% 
    add_metric(list(youden))
#> Assuming the positive class is yes
#> Assuming the positive class has higher x values
bind_rows(cp1, cp2)
#> # A tibble: 2 x 17
#>   direction optimal_cutpoint method            youden      acc sensitivity
#>   <chr>                <dbl> <chr>              <dbl>    <dbl>       <dbl>
#> 1 >=                       2 maximize_metric 0.751792 0.864662    0.888889
#> 2 >=                       6 maximize_metric 0.432348 0.951128    0.444444
#>   specificity      AUC pos_class neg_class prevalence outcome predictor
#>         <dbl>    <dbl> <fct>     <fct>          <dbl> <chr>   <chr>    
#> 1    0.862903 0.923779 yes       no         0.0676692 suicide dsi      
#> 2    0.987903 0.923779 yes       no         0.0676692 suicide dsi      
#>   data               roc_curve                 boot  accuracy
#>   <list>             <list>                    <lgl>    <dbl>
#> 1 <tibble [532 x 2]> <roc_cutpointr [13 x 10]> NA    0.864662
#> 2 <tibble [532 x 2]> <roc_cutpointr [13 x 10]> NA    0.951128

^{Created on 2021-05-05 by the reprex package (v1.0.0)}

[sheilasaia]

@Thie1e Thanks so much for your through reply! I appreciate it and the code you sent along as well. That's very helpful.

[Thie1e]

You're welcome, good to hear that the examples were helpful.