search

Thie1e / cutpointr

Optimal cutpoints in R: determining and validating optimal cutpoints in binary classification
https://cran.r-project.org/package=cutpointr
84 stars 13 forks source link

Doubt about metric sum_sens_spec #29

Closed jgarces02 closed 4 years ago

jgarces02 commented 4 years ago

Hi (again ::sweat::) @Thie1e,

I've a (possibly silly) question, if you don't mind, please. I ran cutpointr(..., metric = sum_sens_spec) and I don't understand how it calculates this cutoff because, when I do it manually my results, are a bit different.

Over the ct$roc_curve I have added a new col (ss) with sens + spec calculation (just for test what I'm saying)...

# A tibble: 379 x 10
   x.sorted    tp    fp    tn    fn     tpr   tnr   fpr   fnr    ss
      <dbl> <dbl> <dbl> <int> <int>   <dbl> <dbl> <dbl> <dbl> <dbl>
 1   Inf        0     0    29   349 0           1     0 1      1   
 2     6.32     1     0    29   348 0.00287     1     0 0.997  1.00
 3     5.17     2     0    29   347 0.00573     1     0 0.994  1.01
 4     4.88     3     0    29   346 0.00860     1     0 0.991  1.01
 5     4.78     4     0    29   345 0.0115      1     0 0.989  1.01
 6     4.47     5     0    29   344 0.0143      1     0 0.986  1.01
 7     4.27     6     0    29   343 0.0172      1     0 0.983  1.02
 8     4.12     7     0    29   342 0.0201      1     0 0.980  1.02
 9     4.00     8     0    29   341 0.0229      1     0 0.977  1.02
10     3.85     9     0    29   340 0.0258      1     0 0.974  1.03
# ... with 369 more rows

... and the x.sorted with higher value is distinct the one calculated by cutpointr. What I'm missing, please?

Thie1e commented 4 years ago

Hi,

can you perhaps make this problem reproducible by posting the complete data and code? Or some subset of the data that illustrates the problem? If the data is rather large, you can put dput(mydata) on pastebin, for example.

My guess is that there were multiple optimal cutpoints and the median was returned, if the parameter break_ties was not changed. There's not much I can tell by the first lines of the ROC curve.

jgarces02 commented 4 years ago

Yes, I guess that a head is not enough, it was just in case...

structure(list(x.sorted = c(Inf, 6.315263451, 5.170684444, 4.877609364, 
4.782353157, 4.474549463, 4.273921089, 4.122465156, 4.000705813, 
3.854426206, 3.6928992, 3.678133407, 3.627976393, 3.581001391, 
3.483279444, 3.378620952, 3.372424961, 3.269772615, 3.207061971, 
3.204966446, 3.177320725, 3.163491212, 2.986489972, 2.921139063, 
2.8959012, 2.867251471, 2.861370101, 2.806651647, 2.704984, 2.648089572, 
2.619723224, 2.596228666, 2.544907904, 2.505740351, 2.422510092, 
2.39897944, 2.3283052, 2.292259404, 2.259753031, 2.207856671, 
2.179837442, 2.154146987, 2.125988864, 2.104957586, 2.104893365, 
2.101455961, 2.093432707, 2.088926599, 2.078215915, 2.061478525, 
2.027699662, 1.964978076, 1.934153913, 1.918243059, 1.901591361, 
1.873115523, 1.859677384, 1.848990598, 1.847506402, 1.838536537, 
1.802418426, 1.800451027, 1.796107849, 1.793267533, 1.790135689, 
1.785559388, 1.755635745, 1.727025904, 1.703292172, 1.658919618, 
1.61292873, 1.612726949, 1.59711084, 1.594956892, 1.5948724, 
1.593759152, 1.573824122, 1.562346405, 1.552325413, 1.552107853, 
1.549803922, 1.548718385, 1.524129314, 1.5073218, 1.502104347, 
1.487163317, 1.486159001, 1.485470064, 1.459023256, 1.451937716, 
1.44652186, 1.41474872, 1.401337584, 1.39363337, 1.391494495, 
1.383406677, 1.382764706, 1.379917595, 1.355311495, 1.34527235, 
1.344956198, 1.342642, 1.336793635, 1.333386655, 1.328632154, 
1.326216393, 1.325906684, 1.325504301, 1.325049575, 1.322017723, 
1.3195044, 1.313179027, 1.308044359, 1.301685884, 1.300331465, 
1.277128459, 1.273090703, 1.25776591, 1.250565147, 1.246608088, 
1.245354314, 1.23130497, 1.229318862, 1.223767768, 1.219780837, 
1.203859157, 1.201843237, 1.178914564, 1.175472204, 1.174959781, 
1.16844, 1.165591746, 1.160866361, 1.156344584, 1.155033016, 
1.145289458, 1.142560783, 1.138071211, 1.125781074, 1.119211986, 
1.111947865, 1.105618572, 1.102241574, 1.092799218, 1.088284451, 
1.085129773, 1.0717914, 1.070167002, 1.060884, 1.0605446, 1.058145872, 
1.057855315, 1.034757318, 1.026408706, 1.02531048, 1.0245624, 
1.011012679, 1.007623164, 1.002940042, 0.999237837, 0.990686071, 
0.9823608, 0.963732988, 0.962268926, 0.954027274, 0.950661866, 
0.944094256, 0.942658, 0.931368509, 0.929468496, 0.928504441, 
0.924808598, 0.912367059, 0.901732919, 0.899920212, 0.891769529, 
0.886020471, 0.878704284, 0.876812113, 0.870450678, 0.869241947, 
0.862239885, 0.8584866, 0.8381508, 0.832701108, 0.829326053, 
0.822431373, 0.814420233, 0.811750615, 0.807409037, 0.803801644, 
0.803750125, 0.795526593, 0.794179529, 0.779449567, 0.770609031, 
0.768485463, 0.752808719, 0.747698906, 0.742881311, 0.742482168, 
0.73826484, 0.734165922, 0.733195257, 0.732633129, 0.725428651, 
0.724738138, 0.723802111, 0.723130112, 0.718790368, 0.7161632, 
0.710050411, 0.7062392, 0.704181066, 0.704082, 0.702871754, 0.7007556, 
0.698108232, 0.69397588, 0.68813539, 0.684843769, 0.681886429, 
0.674630066, 0.668878433, 0.668379918, 0.667625951, 0.665277422, 
0.659985407, 0.655709318, 0.6529344, 0.650411915, 0.649132663, 
0.647367744, 0.644018957, 0.635013993, 0.633368676, 0.631073772, 
0.613420732, 0.612176864, 0.611610606, 0.609460615, 0.598011813, 
0.5937282, 0.591944154, 0.587068247, 0.58525636, 0.582862, 0.581956274, 
0.5815292, 0.581022, 0.579939592, 0.572809377, 0.568451752, 0.562399593, 
0.562061794, 0.55372792, 0.548372329, 0.547819268, 0.5473626, 
0.542116277, 0.541948911, 0.541266664, 0.541192825, 0.5386198, 
0.534743996, 0.5339088, 0.516795058, 0.515100223, 0.513132966, 
0.502413793, 0.499747225, 0.493226275, 0.491471638, 0.476123212, 
0.4731648, 0.472882577, 0.471947895, 0.470937878, 0.466512122, 
0.465773157, 0.460947442, 0.451086781, 0.45105227, 0.444123224, 
0.441478162, 0.44088182, 0.435961112, 0.435397578, 0.434402963, 
0.431761624, 0.42584029, 0.4217052, 0.414179788, 0.401326243, 
0.399400772, 0.395851057, 0.392769529, 0.3895648, 0.381949202, 
0.381422825, 0.381258652, 0.378531996, 0.376691741, 0.374340167, 
0.372506453, 0.371187164, 0.368529428, 0.366833566, 0.365212098, 
0.361758829, 0.360472717, 0.358903239, 0.343563485, 0.341633632, 
0.339604278, 0.338816804, 0.338709576, 0.329576961, 0.329036654, 
0.324155192, 0.32112164, 0.31896413, 0.314625386, 0.312928, 0.309803155, 
0.306768374, 0.303039145, 0.296370629, 0.290155111, 0.2900744, 
0.28833279, 0.283764932, 0.282674387, 0.282649612, 0.279056764, 
0.269359646, 0.256732633, 0.249303806, 0.24194598, 0.241791152, 
0.240749772, 0.239731921, 0.232869349, 0.22439472, 0.217397289, 
0.212104694, 0.209388233, 0.209163874, 0.204238777, 0.203882044, 
0.201730616, 0.193840766, 0.19032, 0.1888292, 0.187437048, 0.184875782, 
0.180033433, 0.171027241, 0.164005362, 0.154581098, 0.153733804, 
0.1384584, 0.13667467, 0.1176252, 0.113781, 0.09824986, 0.088172507, 
0.078665313, 0.077467691, 0.075028396, 0.056924362, 0.052317956, 
0.0502272, 0.037646946, 0.020823606, 0.009356919, 0.0083304, 
0.00645039, 0), tp = c(0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 
12, 13, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 
27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 
43, 44, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 
58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 
74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 
90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 
105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 
118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 
131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 
144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 
157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 
169, 170, 171, 172, 173, 174, 175, 176, 176, 177, 178, 179, 180, 
181, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 191, 
192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 
205, 205, 206, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 
216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 
229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 240, 
241, 241, 242, 243, 243, 244, 245, 246, 246, 247, 248, 249, 250, 
251, 252, 253, 254, 255, 256, 257, 258, 259, 260, 261, 262, 263, 
263, 264, 265, 266, 267, 268, 269, 270, 271, 272, 272, 273, 274, 
275, 276, 277, 278, 279, 280, 281, 282, 283, 284, 285, 286, 287, 
288, 289, 290, 291, 292, 292, 293, 294, 295, 296, 297, 297, 298, 
299, 300, 301, 302, 303, 304, 305, 306, 307, 307, 308, 309, 310, 
311, 312, 313, 314, 315, 316, 316, 317, 318, 319, 320, 321, 322, 
323, 324, 325, 326, 327, 328, 329, 330, 331, 332, 333, 333, 333, 
334, 335, 336, 337, 338, 339, 339, 340, 341, 341, 341, 342, 343, 
343, 343, 344, 345, 345, 346, 346, 347, 348, 348, 349, 349), 
    fp = c(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 
    1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 
    1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 
    2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 
    2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 
    2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 
    2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 
    2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 
    2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 
    2, 2, 2, 3, 3, 3, 3, 3, 3, 3, 3, 4, 4, 4, 4, 4, 4, 5, 5, 
    5, 5, 5, 5, 5, 5, 5, 5, 5, 6, 6, 6, 6, 6, 6, 6, 6, 6, 6, 
    6, 6, 6, 6, 6, 7, 7, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 
    8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 8, 
    8, 8, 8, 8, 9, 9, 10, 10, 10, 11, 11, 11, 11, 12, 12, 12, 
    12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 12, 
    13, 13, 13, 13, 13, 13, 13, 13, 13, 13, 14, 14, 14, 14, 14, 
    14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 
    14, 15, 15, 15, 15, 15, 15, 16, 16, 16, 16, 16, 16, 16, 16, 
    16, 16, 16, 17, 17, 17, 17, 17, 17, 17, 17, 17, 17, 18, 18, 
    18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 
    18, 19, 20, 20, 20, 20, 20, 20, 20, 21, 21, 21, 22, 23, 23, 
    23, 24, 25, 25, 25, 26, 26, 27, 27, 27, 28, 28, 29), tn = c(29L, 
    29L, 29L, 29L, 29L, 29L, 29L, 29L, 29L, 29L, 29L, 29L, 29L, 
    29L, 28L, 28L, 28L, 28L, 28L, 28L, 28L, 28L, 28L, 28L, 28L, 
    28L, 28L, 28L, 28L, 28L, 28L, 28L, 28L, 28L, 28L, 28L, 28L, 
    28L, 28L, 28L, 28L, 28L, 28L, 28L, 28L, 28L, 27L, 27L, 27L, 
    27L, 27L, 27L, 27L, 27L, 27L, 27L, 27L, 27L, 27L, 27L, 27L, 
    27L, 27L, 27L, 27L, 27L, 27L, 27L, 27L, 27L, 27L, 27L, 27L, 
    27L, 27L, 27L, 27L, 27L, 27L, 27L, 27L, 27L, 27L, 27L, 27L, 
    27L, 27L, 27L, 27L, 27L, 27L, 27L, 27L, 27L, 27L, 27L, 27L, 
    27L, 27L, 27L, 27L, 27L, 27L, 27L, 27L, 27L, 27L, 27L, 27L, 
    27L, 27L, 27L, 27L, 27L, 27L, 27L, 27L, 27L, 27L, 27L, 27L, 
    27L, 27L, 27L, 27L, 27L, 27L, 27L, 27L, 27L, 27L, 27L, 27L, 
    27L, 27L, 27L, 27L, 27L, 27L, 27L, 27L, 27L, 27L, 27L, 27L, 
    27L, 27L, 27L, 27L, 27L, 27L, 27L, 27L, 27L, 27L, 27L, 27L, 
    27L, 27L, 27L, 27L, 27L, 27L, 27L, 27L, 27L, 27L, 27L, 27L, 
    27L, 27L, 27L, 26L, 26L, 26L, 26L, 26L, 26L, 26L, 26L, 25L, 
    25L, 25L, 25L, 25L, 25L, 24L, 24L, 24L, 24L, 24L, 24L, 24L, 
    24L, 24L, 24L, 24L, 23L, 23L, 23L, 23L, 23L, 23L, 23L, 23L, 
    23L, 23L, 23L, 23L, 23L, 23L, 23L, 22L, 22L, 21L, 21L, 21L, 
    21L, 21L, 21L, 21L, 21L, 21L, 21L, 21L, 21L, 21L, 21L, 21L, 
    21L, 21L, 21L, 21L, 21L, 21L, 21L, 21L, 21L, 21L, 21L, 21L, 
    21L, 21L, 21L, 21L, 21L, 21L, 21L, 21L, 20L, 20L, 19L, 19L, 
    19L, 18L, 18L, 18L, 18L, 17L, 17L, 17L, 17L, 17L, 17L, 17L, 
    17L, 17L, 17L, 17L, 17L, 17L, 17L, 17L, 17L, 17L, 17L, 16L, 
    16L, 16L, 16L, 16L, 16L, 16L, 16L, 16L, 16L, 15L, 15L, 15L, 
    15L, 15L, 15L, 15L, 15L, 15L, 15L, 15L, 15L, 15L, 15L, 15L, 
    15L, 15L, 15L, 15L, 15L, 15L, 14L, 14L, 14L, 14L, 14L, 14L, 
    13L, 13L, 13L, 13L, 13L, 13L, 13L, 13L, 13L, 13L, 13L, 12L, 
    12L, 12L, 12L, 12L, 12L, 12L, 12L, 12L, 12L, 11L, 11L, 11L, 
    11L, 11L, 11L, 11L, 11L, 11L, 11L, 11L, 11L, 11L, 11L, 11L, 
    11L, 11L, 11L, 10L, 9L, 9L, 9L, 9L, 9L, 9L, 9L, 8L, 8L, 8L, 
    7L, 6L, 6L, 6L, 5L, 4L, 4L, 4L, 3L, 3L, 2L, 2L, 2L, 1L, 1L, 
    0L), fn = c(349L, 348L, 347L, 346L, 345L, 344L, 343L, 342L, 
    341L, 340L, 339L, 338L, 337L, 336L, 336L, 335L, 334L, 333L, 
    332L, 331L, 330L, 329L, 328L, 327L, 326L, 325L, 324L, 323L, 
    322L, 321L, 320L, 319L, 318L, 317L, 316L, 315L, 314L, 313L, 
    312L, 311L, 310L, 309L, 308L, 307L, 306L, 305L, 305L, 304L, 
    303L, 302L, 301L, 300L, 299L, 298L, 297L, 296L, 295L, 294L, 
    293L, 292L, 291L, 290L, 289L, 288L, 287L, 286L, 285L, 284L, 
    283L, 282L, 281L, 280L, 279L, 278L, 277L, 276L, 275L, 274L, 
    273L, 272L, 271L, 270L, 269L, 268L, 267L, 266L, 265L, 264L, 
    263L, 262L, 261L, 260L, 259L, 258L, 257L, 256L, 255L, 254L, 
    253L, 252L, 251L, 250L, 249L, 248L, 247L, 246L, 245L, 244L, 
    243L, 242L, 241L, 240L, 239L, 238L, 237L, 236L, 235L, 234L, 
    233L, 232L, 231L, 230L, 229L, 228L, 227L, 226L, 225L, 224L, 
    223L, 222L, 221L, 220L, 219L, 218L, 217L, 216L, 215L, 214L, 
    213L, 212L, 211L, 210L, 209L, 208L, 207L, 206L, 205L, 204L, 
    203L, 202L, 201L, 200L, 199L, 198L, 197L, 196L, 195L, 194L, 
    193L, 192L, 191L, 190L, 189L, 188L, 187L, 186L, 185L, 184L, 
    183L, 182L, 181L, 180L, 180L, 179L, 178L, 177L, 176L, 175L, 
    174L, 173L, 173L, 172L, 171L, 170L, 169L, 168L, 168L, 167L, 
    166L, 165L, 164L, 163L, 162L, 161L, 160L, 159L, 158L, 158L, 
    157L, 156L, 155L, 154L, 153L, 152L, 151L, 150L, 149L, 148L, 
    147L, 146L, 145L, 144L, 144L, 143L, 143L, 142L, 141L, 140L, 
    139L, 138L, 137L, 136L, 135L, 134L, 133L, 132L, 131L, 130L, 
    129L, 128L, 127L, 126L, 125L, 124L, 123L, 122L, 121L, 120L, 
    119L, 118L, 117L, 116L, 115L, 114L, 113L, 112L, 111L, 110L, 
    109L, 109L, 108L, 108L, 107L, 106L, 106L, 105L, 104L, 103L, 
    103L, 102L, 101L, 100L, 99L, 98L, 97L, 96L, 95L, 94L, 93L, 
    92L, 91L, 90L, 89L, 88L, 87L, 86L, 86L, 85L, 84L, 83L, 82L, 
    81L, 80L, 79L, 78L, 77L, 77L, 76L, 75L, 74L, 73L, 72L, 71L, 
    70L, 69L, 68L, 67L, 66L, 65L, 64L, 63L, 62L, 61L, 60L, 59L, 
    58L, 57L, 57L, 56L, 55L, 54L, 53L, 52L, 52L, 51L, 50L, 49L, 
    48L, 47L, 46L, 45L, 44L, 43L, 42L, 42L, 41L, 40L, 39L, 38L, 
    37L, 36L, 35L, 34L, 33L, 33L, 32L, 31L, 30L, 29L, 28L, 27L, 
    26L, 25L, 24L, 23L, 22L, 21L, 20L, 19L, 18L, 17L, 16L, 16L, 
    16L, 15L, 14L, 13L, 12L, 11L, 10L, 10L, 9L, 8L, 8L, 8L, 7L, 
    6L, 6L, 6L, 5L, 4L, 4L, 3L, 3L, 2L, 1L, 1L, 0L, 0L), tpr = c(0, 
    0.00286532951289398, 0.00573065902578797, 0.00859598853868195, 
    0.0114613180515759, 0.0143266475644699, 0.0171919770773639, 
    0.0200573065902579, 0.0229226361031519, 0.0257879656160458, 
    0.0286532951289398, 0.0315186246418338, 0.0343839541547278, 
    0.0372492836676218, 0.0372492836676218, 0.0401146131805158, 
    0.0429799426934097, 0.0458452722063037, 0.0487106017191977, 
    0.0515759312320917, 0.0544412607449857, 0.0573065902578797, 
    0.0601719197707736, 0.0630372492836676, 0.0659025787965616, 
    0.0687679083094556, 0.0716332378223496, 0.0744985673352436, 
    0.0773638968481375, 0.0802292263610315, 0.0830945558739255, 
    0.0859598853868195, 0.0888252148997135, 0.0916905444126074, 
    0.0945558739255014, 0.0974212034383954, 0.100286532951289, 
    0.103151862464183, 0.106017191977077, 0.108882521489971, 
    0.111747851002865, 0.114613180515759, 0.117478510028653, 
    0.120343839541547, 0.123209169054441, 0.126074498567335, 
    0.126074498567335, 0.128939828080229, 0.131805157593123, 
    0.134670487106017, 0.137535816618911, 0.140401146131805, 
    0.143266475644699, 0.146131805157593, 0.148997134670487, 
    0.151862464183381, 0.154727793696275, 0.157593123209169, 
    0.160458452722063, 0.163323782234957, 0.166189111747851, 
    0.169054441260745, 0.171919770773639, 0.174785100286533, 
    0.177650429799427, 0.180515759312321, 0.183381088825215, 
    0.186246418338109, 0.189111747851003, 0.191977077363897, 
    0.194842406876791, 0.197707736389685, 0.200573065902579, 
    0.203438395415473, 0.206303724928367, 0.209169054441261, 
    0.212034383954155, 0.214899713467049, 0.217765042979943, 
    0.220630372492837, 0.223495702005731, 0.226361031518625, 
    0.229226361031519, 0.232091690544413, 0.234957020057307, 
    0.237822349570201, 0.240687679083095, 0.243553008595989, 
    0.246418338108883, 0.249283667621777, 0.25214899713467, 0.255014326647564, 
    0.257879656160458, 0.260744985673352, 0.263610315186246, 
    0.26647564469914, 0.269340974212034, 0.272206303724928, 0.275071633237822, 
    0.277936962750716, 0.28080229226361, 0.283667621776504, 0.286532951289398, 
    0.289398280802292, 0.292263610315186, 0.29512893982808, 0.297994269340974, 
    0.300859598853868, 0.303724928366762, 0.306590257879656, 
    0.30945558739255, 0.312320916905444, 0.315186246418338, 0.318051575931232, 
    0.320916905444126, 0.32378223495702, 0.326647564469914, 0.329512893982808, 
    0.332378223495702, 0.335243553008596, 0.33810888252149, 0.340974212034384, 
    0.343839541547278, 0.346704871060172, 0.349570200573066, 
    0.35243553008596, 0.355300859598854, 0.358166189111748, 0.361031518624642, 
    0.363896848137536, 0.36676217765043, 0.369627507163324, 0.372492836676218, 
    0.375358166189112, 0.378223495702006, 0.3810888252149, 0.383954154727794, 
    0.386819484240688, 0.389684813753582, 0.392550143266476, 
    0.39541547277937, 0.398280802292264, 0.401146131805158, 0.404011461318052, 
    0.406876790830946, 0.40974212034384, 0.412607449856734, 0.415472779369628, 
    0.418338108882521, 0.421203438395415, 0.424068767908309, 
    0.426934097421203, 0.429799426934097, 0.432664756446991, 
    0.435530085959885, 0.438395415472779, 0.441260744985673, 
    0.444126074498567, 0.446991404011461, 0.449856733524355, 
    0.452722063037249, 0.455587392550143, 0.458452722063037, 
    0.461318051575931, 0.464183381088825, 0.467048710601719, 
    0.469914040114613, 0.472779369627507, 0.475644699140401, 
    0.478510028653295, 0.481375358166189, 0.484240687679083, 
    0.484240687679083, 0.487106017191977, 0.489971346704871, 
    0.492836676217765, 0.495702005730659, 0.498567335243553, 
    0.501432664756447, 0.504297994269341, 0.504297994269341, 
    0.507163323782235, 0.510028653295129, 0.512893982808023, 
    0.515759312320917, 0.518624641833811, 0.518624641833811, 
    0.521489971346705, 0.524355300859599, 0.527220630372493, 
    0.530085959885387, 0.532951289398281, 0.535816618911175, 
    0.538681948424069, 0.541547277936963, 0.544412607449857, 
    0.547277936962751, 0.547277936962751, 0.550143266475645, 
    0.553008595988539, 0.555873925501433, 0.558739255014327, 
    0.561604584527221, 0.564469914040115, 0.567335243553009, 
    0.570200573065903, 0.573065902578797, 0.575931232091691, 
    0.578796561604584, 0.581661891117478, 0.584527220630373, 
    0.587392550143267, 0.587392550143267, 0.590257879656161, 
    0.590257879656161, 0.593123209169054, 0.595988538681948, 
    0.598853868194842, 0.601719197707736, 0.60458452722063, 0.607449856733524, 
    0.610315186246418, 0.613180515759312, 0.616045845272206, 
    0.6189111747851, 0.621776504297994, 0.624641833810888, 0.627507163323782, 
    0.630372492836676, 0.63323782234957, 0.636103151862464, 0.638968481375358, 
    0.641833810888252, 0.644699140401146, 0.64756446991404, 0.650429799426934, 
    0.653295128939828, 0.656160458452722, 0.659025787965616, 
    0.66189111747851, 0.664756446991404, 0.667621776504298, 0.670487106017192, 
    0.673352435530086, 0.67621776504298, 0.679083094555874, 0.681948424068768, 
    0.684813753581662, 0.687679083094556, 0.687679083094556, 
    0.69054441260745, 0.69054441260745, 0.693409742120344, 0.696275071633238, 
    0.696275071633238, 0.699140401146132, 0.702005730659026, 
    0.70487106017192, 0.70487106017192, 0.707736389684814, 0.710601719197708, 
    0.713467048710602, 0.716332378223496, 0.71919770773639, 0.722063037249284, 
    0.724928366762178, 0.727793696275072, 0.730659025787966, 
    0.73352435530086, 0.736389684813754, 0.739255014326648, 0.742120343839542, 
    0.744985673352435, 0.747851002865329, 0.750716332378224, 
    0.753581661891118, 0.753581661891118, 0.756446991404012, 
    0.759312320916905, 0.762177650429799, 0.765042979942693, 
    0.767908309455587, 0.770773638968481, 0.773638968481375, 
    0.776504297994269, 0.779369627507163, 0.779369627507163, 
    0.782234957020057, 0.785100286532951, 0.787965616045845, 
    0.790830945558739, 0.793696275071633, 0.796561604584527, 
    0.799426934097421, 0.802292263610315, 0.805157593123209, 
    0.808022922636103, 0.810888252148997, 0.813753581661891, 
    0.816618911174785, 0.819484240687679, 0.822349570200573, 
    0.825214899713467, 0.828080229226361, 0.830945558739255, 
    0.833810888252149, 0.836676217765043, 0.836676217765043, 
    0.839541547277937, 0.842406876790831, 0.845272206303725, 
    0.848137535816619, 0.851002865329513, 0.851002865329513, 
    0.853868194842407, 0.856733524355301, 0.859598853868195, 
    0.862464183381089, 0.865329512893983, 0.868194842406877, 
    0.871060171919771, 0.873925501432665, 0.876790830945559, 
    0.879656160458453, 0.879656160458453, 0.882521489971347, 
    0.885386819484241, 0.888252148997135, 0.891117478510029, 
    0.893982808022923, 0.896848137535817, 0.899713467048711, 
    0.902578796561605, 0.905444126074499, 0.905444126074499, 
    0.908309455587393, 0.911174785100286, 0.91404011461318, 0.916905444126075, 
    0.919770773638969, 0.922636103151863, 0.925501432664756, 
    0.92836676217765, 0.931232091690544, 0.934097421203438, 0.936962750716332, 
    0.939828080229226, 0.94269340974212, 0.945558739255014, 0.948424068767908, 
    0.951289398280802, 0.954154727793696, 0.954154727793696, 
    0.954154727793696, 0.95702005730659, 0.959885386819484, 0.962750716332378, 
    0.965616045845272, 0.968481375358166, 0.97134670487106, 0.97134670487106, 
    0.974212034383954, 0.977077363896848, 0.977077363896848, 
    0.977077363896848, 0.979942693409742, 0.982808022922636, 
    0.982808022922636, 0.982808022922636, 0.98567335243553, 0.988538681948424, 
    0.988538681948424, 0.991404011461318, 0.991404011461318, 
    0.994269340974212, 0.997134670487106, 0.997134670487106, 
    1, 1), tnr = c(1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 
    0.96551724137931, 0.96551724137931, 0.96551724137931, 0.96551724137931, 
    0.96551724137931, 0.96551724137931, 0.96551724137931, 0.96551724137931, 
    0.96551724137931, 0.96551724137931, 0.96551724137931, 0.96551724137931, 
    0.96551724137931, 0.96551724137931, 0.96551724137931, 0.96551724137931, 
    0.96551724137931, 0.96551724137931, 0.96551724137931, 0.96551724137931, 
    0.96551724137931, 0.96551724137931, 0.96551724137931, 0.96551724137931, 
    0.96551724137931, 0.96551724137931, 0.96551724137931, 0.96551724137931, 
    0.96551724137931, 0.96551724137931, 0.96551724137931, 0.96551724137931, 
    0.931034482758621, 0.931034482758621, 0.931034482758621, 
    0.931034482758621, 0.931034482758621, 0.931034482758621, 
    0.931034482758621, 0.931034482758621, 0.931034482758621, 
    0.931034482758621, 0.931034482758621, 0.931034482758621, 
    0.931034482758621, 0.931034482758621, 0.931034482758621, 
    0.931034482758621, 0.931034482758621, 0.931034482758621, 
    0.931034482758621, 0.931034482758621, 0.931034482758621, 
    0.931034482758621, 0.931034482758621, 0.931034482758621, 
    0.931034482758621, 0.931034482758621, 0.931034482758621, 
    0.931034482758621, 0.931034482758621, 0.931034482758621, 
    0.931034482758621, 0.931034482758621, 0.931034482758621, 
    0.931034482758621, 0.931034482758621, 0.931034482758621, 
    0.931034482758621, 0.931034482758621, 0.931034482758621, 
    0.931034482758621, 0.931034482758621, 0.931034482758621, 
    0.931034482758621, 0.931034482758621, 0.931034482758621, 
    0.931034482758621, 0.931034482758621, 0.931034482758621, 
    0.931034482758621, 0.931034482758621, 0.931034482758621, 
    0.931034482758621, 0.931034482758621, 0.931034482758621, 
    0.931034482758621, 0.931034482758621, 0.931034482758621, 
    0.931034482758621, 0.931034482758621, 0.931034482758621, 
    0.931034482758621, 0.931034482758621, 0.931034482758621, 
    0.931034482758621, 0.931034482758621, 0.931034482758621, 
    0.931034482758621, 0.931034482758621, 0.931034482758621, 
    0.931034482758621, 0.931034482758621, 0.931034482758621, 
    0.931034482758621, 0.931034482758621, 0.931034482758621, 
    0.931034482758621, 0.931034482758621, 0.931034482758621, 
    0.931034482758621, 0.931034482758621, 0.931034482758621, 
    0.931034482758621, 0.931034482758621, 0.931034482758621, 
    0.931034482758621, 0.931034482758621, 0.931034482758621, 
    0.931034482758621, 0.931034482758621, 0.931034482758621, 
    0.931034482758621, 0.931034482758621, 0.931034482758621, 
    0.931034482758621, 0.931034482758621, 0.931034482758621, 
    0.931034482758621, 0.931034482758621, 0.931034482758621, 
    0.931034482758621, 0.931034482758621, 0.931034482758621, 
    0.931034482758621, 0.931034482758621, 0.931034482758621, 
    0.931034482758621, 0.931034482758621, 0.931034482758621, 
    0.931034482758621, 0.931034482758621, 0.931034482758621, 
    0.931034482758621, 0.931034482758621, 0.931034482758621, 
    0.931034482758621, 0.931034482758621, 0.931034482758621, 
    0.931034482758621, 0.931034482758621, 0.931034482758621, 
    0.931034482758621, 0.931034482758621, 0.931034482758621, 
    0.931034482758621, 0.931034482758621, 0.931034482758621, 
    0.896551724137931, 0.896551724137931, 0.896551724137931, 
    0.896551724137931, 0.896551724137931, 0.896551724137931, 
    0.896551724137931, 0.896551724137931, 0.862068965517241, 
    0.862068965517241, 0.862068965517241, 0.862068965517241, 
    0.862068965517241, 0.862068965517241, 0.827586206896552, 
    0.827586206896552, 0.827586206896552, 0.827586206896552, 
    0.827586206896552, 0.827586206896552, 0.827586206896552, 
    0.827586206896552, 0.827586206896552, 0.827586206896552, 
    0.827586206896552, 0.793103448275862, 0.793103448275862, 
    0.793103448275862, 0.793103448275862, 0.793103448275862, 
    0.793103448275862, 0.793103448275862, 0.793103448275862, 
    0.793103448275862, 0.793103448275862, 0.793103448275862, 
    0.793103448275862, 0.793103448275862, 0.793103448275862, 
    0.793103448275862, 0.758620689655172, 0.758620689655172, 
    0.724137931034483, 0.724137931034483, 0.724137931034483, 
    0.724137931034483, 0.724137931034483, 0.724137931034483, 
    0.724137931034483, 0.724137931034483, 0.724137931034483, 
    0.724137931034483, 0.724137931034483, 0.724137931034483, 
    0.724137931034483, 0.724137931034483, 0.724137931034483, 
    0.724137931034483, 0.724137931034483, 0.724137931034483, 
    0.724137931034483, 0.724137931034483, 0.724137931034483, 
    0.724137931034483, 0.724137931034483, 0.724137931034483, 
    0.724137931034483, 0.724137931034483, 0.724137931034483, 
    0.724137931034483, 0.724137931034483, 0.724137931034483, 
    0.724137931034483, 0.724137931034483, 0.724137931034483, 
    0.724137931034483, 0.724137931034483, 0.689655172413793, 
    0.689655172413793, 0.655172413793103, 0.655172413793103, 
    0.655172413793103, 0.620689655172414, 0.620689655172414, 
    0.620689655172414, 0.620689655172414, 0.586206896551724, 
    0.586206896551724, 0.586206896551724, 0.586206896551724, 
    0.586206896551724, 0.586206896551724, 0.586206896551724, 
    0.586206896551724, 0.586206896551724, 0.586206896551724, 
    0.586206896551724, 0.586206896551724, 0.586206896551724, 
    0.586206896551724, 0.586206896551724, 0.586206896551724, 
    0.586206896551724, 0.586206896551724, 0.551724137931034, 
    0.551724137931034, 0.551724137931034, 0.551724137931034, 
    0.551724137931034, 0.551724137931034, 0.551724137931034, 
    0.551724137931034, 0.551724137931034, 0.551724137931034, 
    0.517241379310345, 0.517241379310345, 0.517241379310345, 
    0.517241379310345, 0.517241379310345, 0.517241379310345, 
    0.517241379310345, 0.517241379310345, 0.517241379310345, 
    0.517241379310345, 0.517241379310345, 0.517241379310345, 
    0.517241379310345, 0.517241379310345, 0.517241379310345, 
    0.517241379310345, 0.517241379310345, 0.517241379310345, 
    0.517241379310345, 0.517241379310345, 0.517241379310345, 
    0.482758620689655, 0.482758620689655, 0.482758620689655, 
    0.482758620689655, 0.482758620689655, 0.482758620689655, 
    0.448275862068966, 0.448275862068966, 0.448275862068966, 
    0.448275862068966, 0.448275862068966, 0.448275862068966, 
    0.448275862068966, 0.448275862068966, 0.448275862068966, 
    0.448275862068966, 0.448275862068966, 0.413793103448276, 
    0.413793103448276, 0.413793103448276, 0.413793103448276, 
    0.413793103448276, 0.413793103448276, 0.413793103448276, 
    0.413793103448276, 0.413793103448276, 0.413793103448276, 
    0.379310344827586, 0.379310344827586, 0.379310344827586, 
    0.379310344827586, 0.379310344827586, 0.379310344827586, 
    0.379310344827586, 0.379310344827586, 0.379310344827586, 
    0.379310344827586, 0.379310344827586, 0.379310344827586, 
    0.379310344827586, 0.379310344827586, 0.379310344827586, 
    0.379310344827586, 0.379310344827586, 0.379310344827586, 
    0.344827586206897, 0.310344827586207, 0.310344827586207, 
    0.310344827586207, 0.310344827586207, 0.310344827586207, 
    0.310344827586207, 0.310344827586207, 0.275862068965517, 
    0.275862068965517, 0.275862068965517, 0.241379310344828, 
    0.206896551724138, 0.206896551724138, 0.206896551724138, 
    0.172413793103448, 0.137931034482759, 0.137931034482759, 
    0.137931034482759, 0.103448275862069, 0.103448275862069, 
    0.0689655172413793, 0.0689655172413793, 0.0689655172413793, 
    0.0344827586206897, 0.0344827586206897, 0), fpr = c(0, 0, 
    0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.0344827586206896, 0.0344827586206896, 
    0.0344827586206896, 0.0344827586206896, 0.0344827586206896, 
    0.0344827586206896, 0.0344827586206896, 0.0344827586206896, 
    0.0344827586206896, 0.0344827586206896, 0.0344827586206896, 
    0.0344827586206896, 0.0344827586206896, 0.0344827586206896, 
    0.0344827586206896, 0.0344827586206896, 0.0344827586206896, 
    0.0344827586206896, 0.0344827586206896, 0.0344827586206896, 
    0.0344827586206896, 0.0344827586206896, 0.0344827586206896, 
    0.0344827586206896, 0.0344827586206896, 0.0344827586206896, 
    0.0344827586206896, 0.0344827586206896, 0.0344827586206896, 
    0.0344827586206896, 0.0344827586206896, 0.0344827586206896, 
    0.0689655172413793, 0.0689655172413793, 0.0689655172413793, 
    0.0689655172413793, 0.0689655172413793, 0.0689655172413793, 
    0.0689655172413793, 0.0689655172413793, 0.0689655172413793, 
    0.0689655172413793, 0.0689655172413793, 0.0689655172413793, 
    0.0689655172413793, 0.0689655172413793, 0.0689655172413793, 
    0.0689655172413793, 0.0689655172413793, 0.0689655172413793, 
    0.0689655172413793, 0.0689655172413793, 0.0689655172413793, 
    0.0689655172413793, 0.0689655172413793, 0.0689655172413793, 
    0.0689655172413793, 0.0689655172413793, 0.0689655172413793, 
    0.0689655172413793, 0.0689655172413793, 0.0689655172413793, 
    0.0689655172413793, 0.0689655172413793, 0.0689655172413793, 
    0.0689655172413793, 0.0689655172413793, 0.0689655172413793, 
    0.0689655172413793, 0.0689655172413793, 0.0689655172413793, 
    0.0689655172413793, 0.0689655172413793, 0.0689655172413793, 
    0.0689655172413793, 0.0689655172413793, 0.0689655172413793, 
    0.0689655172413793, 0.0689655172413793, 0.0689655172413793, 
    0.0689655172413793, 0.0689655172413793, 0.0689655172413793, 
    0.0689655172413793, 0.0689655172413793, 0.0689655172413793, 
    0.0689655172413793, 0.0689655172413793, 0.0689655172413793, 
    0.0689655172413793, 0.0689655172413793, 0.0689655172413793, 
    0.0689655172413793, 0.0689655172413793, 0.0689655172413793, 
    0.0689655172413793, 0.0689655172413793, 0.0689655172413793, 
    0.0689655172413793, 0.0689655172413793, 0.0689655172413793, 
    0.0689655172413793, 0.0689655172413793, 0.0689655172413793, 
    0.0689655172413793, 0.0689655172413793, 0.0689655172413793, 
    0.0689655172413793, 0.0689655172413793, 0.0689655172413793, 
    0.0689655172413793, 0.0689655172413793, 0.0689655172413793, 
    0.0689655172413793, 0.0689655172413793, 0.0689655172413793, 
    0.0689655172413793, 0.0689655172413793, 0.0689655172413793, 
    0.0689655172413793, 0.0689655172413793, 0.0689655172413793, 
    0.0689655172413793, 0.0689655172413793, 0.0689655172413793, 
    0.0689655172413793, 0.0689655172413793, 0.0689655172413793, 
    0.0689655172413793, 0.0689655172413793, 0.0689655172413793, 
    0.0689655172413793, 0.0689655172413793, 0.0689655172413793, 
    0.0689655172413793, 0.0689655172413793, 0.0689655172413793, 
    0.0689655172413793, 0.0689655172413793, 0.0689655172413793, 
    0.0689655172413793, 0.0689655172413793, 0.0689655172413793, 
    0.0689655172413793, 0.0689655172413793, 0.0689655172413793, 
    0.0689655172413793, 0.0689655172413793, 0.0689655172413793, 
    0.0689655172413793, 0.0689655172413793, 0.0689655172413793, 
    0.0689655172413793, 0.0689655172413793, 0.0689655172413793, 
    0.0689655172413793, 0.0689655172413793, 0.0689655172413793, 
    0.103448275862069, 0.103448275862069, 0.103448275862069, 
    0.103448275862069, 0.103448275862069, 0.103448275862069, 
    0.103448275862069, 0.103448275862069, 0.137931034482759, 
    0.137931034482759, 0.137931034482759, 0.137931034482759, 
    0.137931034482759, 0.137931034482759, 0.172413793103448, 
    0.172413793103448, 0.172413793103448, 0.172413793103448, 
    0.172413793103448, 0.172413793103448, 0.172413793103448, 
    0.172413793103448, 0.172413793103448, 0.172413793103448, 
    0.172413793103448, 0.206896551724138, 0.206896551724138, 
    0.206896551724138, 0.206896551724138, 0.206896551724138, 
    0.206896551724138, 0.206896551724138, 0.206896551724138, 
    0.206896551724138, 0.206896551724138, 0.206896551724138, 
    0.206896551724138, 0.206896551724138, 0.206896551724138, 
    0.206896551724138, 0.241379310344828, 0.241379310344828, 
    0.275862068965517, 0.275862068965517, 0.275862068965517, 
    0.275862068965517, 0.275862068965517, 0.275862068965517, 
    0.275862068965517, 0.275862068965517, 0.275862068965517, 
    0.275862068965517, 0.275862068965517, 0.275862068965517, 
    0.275862068965517, 0.275862068965517, 0.275862068965517, 
    0.275862068965517, 0.275862068965517, 0.275862068965517, 
    0.275862068965517, 0.275862068965517, 0.275862068965517, 
    0.275862068965517, 0.275862068965517, 0.275862068965517, 
    0.275862068965517, 0.275862068965517, 0.275862068965517, 
    0.275862068965517, 0.275862068965517, 0.275862068965517, 
    0.275862068965517, 0.275862068965517, 0.275862068965517, 
    0.275862068965517, 0.275862068965517, 0.310344827586207, 
    0.310344827586207, 0.344827586206897, 0.344827586206897, 
    0.344827586206897, 0.379310344827586, 0.379310344827586, 
    0.379310344827586, 0.379310344827586, 0.413793103448276, 
    0.413793103448276, 0.413793103448276, 0.413793103448276, 
    0.413793103448276, 0.413793103448276, 0.413793103448276, 
    0.413793103448276, 0.413793103448276, 0.413793103448276, 
    0.413793103448276, 0.413793103448276, 0.413793103448276, 
    0.413793103448276, 0.413793103448276, 0.413793103448276, 
    0.413793103448276, 0.413793103448276, 0.448275862068966, 
    0.448275862068966, 0.448275862068966, 0.448275862068966, 
    0.448275862068966, 0.448275862068966, 0.448275862068966, 
    0.448275862068966, 0.448275862068966, 0.448275862068966, 
    0.482758620689655, 0.482758620689655, 0.482758620689655, 
    0.482758620689655, 0.482758620689655, 0.482758620689655, 
    0.482758620689655, 0.482758620689655, 0.482758620689655, 
    0.482758620689655, 0.482758620689655, 0.482758620689655, 
    0.482758620689655, 0.482758620689655, 0.482758620689655, 
    0.482758620689655, 0.482758620689655, 0.482758620689655, 
    0.482758620689655, 0.482758620689655, 0.482758620689655, 
    0.517241379310345, 0.517241379310345, 0.517241379310345, 
    0.517241379310345, 0.517241379310345, 0.517241379310345, 
    0.551724137931034, 0.551724137931034, 0.551724137931034, 
    0.551724137931034, 0.551724137931034, 0.551724137931034, 
    0.551724137931034, 0.551724137931034, 0.551724137931034, 
    0.551724137931034, 0.551724137931034, 0.586206896551724, 
    0.586206896551724, 0.586206896551724, 0.586206896551724, 
    0.586206896551724, 0.586206896551724, 0.586206896551724, 
    0.586206896551724, 0.586206896551724, 0.586206896551724, 
    0.620689655172414, 0.620689655172414, 0.620689655172414, 
    0.620689655172414, 0.620689655172414, 0.620689655172414, 
    0.620689655172414, 0.620689655172414, 0.620689655172414, 
    0.620689655172414, 0.620689655172414, 0.620689655172414, 
    0.620689655172414, 0.620689655172414, 0.620689655172414, 
    0.620689655172414, 0.620689655172414, 0.620689655172414, 
    0.655172413793103, 0.689655172413793, 0.689655172413793, 
    0.689655172413793, 0.689655172413793, 0.689655172413793, 
    0.689655172413793, 0.689655172413793, 0.724137931034483, 
    0.724137931034483, 0.724137931034483, 0.758620689655172, 
    0.793103448275862, 0.793103448275862, 0.793103448275862, 
    0.827586206896552, 0.862068965517241, 0.862068965517241, 
    0.862068965517241, 0.896551724137931, 0.896551724137931, 
    0.931034482758621, 0.931034482758621, 0.931034482758621, 
    0.96551724137931, 0.96551724137931, 1), fnr = c(1, 0.997134670487106, 
    0.994269340974212, 0.991404011461318, 0.988538681948424, 
    0.98567335243553, 0.982808022922636, 0.979942693409742, 0.977077363896848, 
    0.974212034383954, 0.97134670487106, 0.968481375358166, 0.965616045845272, 
    0.962750716332378, 0.962750716332378, 0.959885386819484, 
    0.95702005730659, 0.954154727793696, 0.951289398280802, 0.948424068767908, 
    0.945558739255014, 0.94269340974212, 0.939828080229226, 0.936962750716332, 
    0.934097421203438, 0.931232091690544, 0.92836676217765, 0.925501432664756, 
    0.922636103151863, 0.919770773638968, 0.916905444126074, 
    0.91404011461318, 0.911174785100286, 0.908309455587393, 0.905444126074499, 
    0.902578796561605, 0.899713467048711, 0.896848137535817, 
    0.893982808022923, 0.891117478510029, 0.888252148997135, 
    0.885386819484241, 0.882521489971347, 0.879656160458453, 
    0.876790830945559, 0.873925501432665, 0.873925501432665, 
    0.871060171919771, 0.868194842406877, 0.865329512893983, 
    0.862464183381089, 0.859598853868195, 0.856733524355301, 
    0.853868194842407, 0.851002865329513, 0.848137535816619, 
    0.845272206303725, 0.842406876790831, 0.839541547277937, 
    0.836676217765043, 0.833810888252149, 0.830945558739255, 
    0.828080229226361, 0.825214899713467, 0.822349570200573, 
    0.819484240687679, 0.816618911174785, 0.813753581661891, 
    0.810888252148997, 0.808022922636103, 0.805157593123209, 
    0.802292263610315, 0.799426934097421, 0.796561604584527, 
    0.793696275071633, 0.790830945558739, 0.787965616045845, 
    0.785100286532951, 0.782234957020057, 0.779369627507163, 
    0.776504297994269, 0.773638968481375, 0.770773638968481, 
    0.767908309455587, 0.765042979942693, 0.762177650429799, 
    0.759312320916905, 0.756446991404012, 0.753581661891118, 
    0.750716332378224, 0.74785100286533, 0.744985673352436, 0.742120343839542, 
    0.739255014326648, 0.736389684813754, 0.73352435530086, 0.730659025787966, 
    0.727793696275072, 0.724928366762178, 0.722063037249284, 
    0.71919770773639, 0.716332378223496, 0.713467048710602, 0.710601719197708, 
    0.707736389684814, 0.70487106017192, 0.702005730659026, 0.699140401146132, 
    0.696275071633238, 0.693409742120344, 0.69054441260745, 0.687679083094556, 
    0.684813753581662, 0.681948424068768, 0.679083094555874, 
    0.67621776504298, 0.673352435530086, 0.670487106017192, 0.667621776504298, 
    0.664756446991404, 0.66189111747851, 0.659025787965616, 0.656160458452722, 
    0.653295128939828, 0.650429799426934, 0.64756446991404, 0.644699140401146, 
    0.641833810888252, 0.638968481375358, 0.636103151862464, 
    0.63323782234957, 0.630372492836676, 0.627507163323782, 0.624641833810888, 
    0.621776504297994, 0.6189111747851, 0.616045845272206, 0.613180515759312, 
    0.610315186246418, 0.607449856733524, 0.60458452722063, 0.601719197707736, 
    0.598853868194842, 0.595988538681948, 0.593123209169054, 
    0.59025787965616, 0.587392550143266, 0.584527220630372, 0.581661891117478, 
    0.578796561604584, 0.575931232091691, 0.573065902578797, 
    0.570200573065903, 0.567335243553009, 0.564469914040115, 
    0.561604584527221, 0.558739255014327, 0.555873925501433, 
    0.553008595988539, 0.550143266475645, 0.547277936962751, 
    0.544412607449857, 0.541547277936963, 0.538681948424069, 
    0.535816618911175, 0.532951289398281, 0.530085959885387, 
    0.527220630372493, 0.524355300859599, 0.521489971346705, 
    0.518624641833811, 0.515759312320917, 0.515759312320917, 
    0.512893982808023, 0.510028653295129, 0.507163323782235, 
    0.504297994269341, 0.501432664756447, 0.498567335243553, 
    0.495702005730659, 0.495702005730659, 0.492836676217765, 
    0.489971346704871, 0.487106017191977, 0.484240687679083, 
    0.481375358166189, 0.481375358166189, 0.478510028653295, 
    0.475644699140401, 0.472779369627507, 0.469914040114613, 
    0.467048710601719, 0.464183381088825, 0.461318051575931, 
    0.458452722063037, 0.455587392550143, 0.452722063037249, 
    0.452722063037249, 0.449856733524355, 0.446991404011461, 
    0.444126074498567, 0.441260744985673, 0.438395415472779, 
    0.435530085959885, 0.432664756446991, 0.429799426934097, 
    0.426934097421203, 0.424068767908309, 0.421203438395416, 
    0.418338108882522, 0.415472779369627, 0.412607449856733, 
    0.412607449856733, 0.409742120343839, 0.409742120343839, 
    0.406876790830946, 0.404011461318052, 0.401146131805158, 
    0.398280802292264, 0.39541547277937, 0.392550143266476, 0.389684813753582, 
    0.386819484240688, 0.383954154727794, 0.3810888252149, 0.378223495702006, 
    0.375358166189112, 0.372492836676218, 0.369627507163324, 
    0.36676217765043, 0.363896848137536, 0.361031518624642, 0.358166189111748, 
    0.355300859598854, 0.35243553008596, 0.349570200573066, 0.346704871060172, 
    0.343839541547278, 0.340974212034384, 0.33810888252149, 0.335243553008596, 
    0.332378223495702, 0.329512893982808, 0.326647564469914, 
    0.32378223495702, 0.320916905444126, 0.318051575931232, 0.315186246418338, 
    0.312320916905444, 0.312320916905444, 0.30945558739255, 0.30945558739255, 
    0.306590257879656, 0.303724928366762, 0.303724928366762, 
    0.300859598853868, 0.297994269340974, 0.29512893982808, 0.29512893982808, 
    0.292263610315186, 0.289398280802292, 0.286532951289398, 
    0.283667621776504, 0.28080229226361, 0.277936962750716, 0.275071633237822, 
    0.272206303724928, 0.269340974212034, 0.26647564469914, 0.263610315186246, 
    0.260744985673352, 0.257879656160458, 0.255014326647565, 
    0.252148997134671, 0.249283667621776, 0.246418338108882, 
    0.246418338108882, 0.243553008595988, 0.240687679083095, 
    0.237822349570201, 0.234957020057307, 0.232091690544413, 
    0.229226361031519, 0.226361031518625, 0.223495702005731, 
    0.220630372492837, 0.220630372492837, 0.217765042979943, 
    0.214899713467049, 0.212034383954155, 0.209169054441261, 
    0.206303724928367, 0.203438395415473, 0.200573065902579, 
    0.197707736389685, 0.194842406876791, 0.191977077363897, 
    0.189111747851003, 0.186246418338109, 0.183381088825215, 
    0.180515759312321, 0.177650429799427, 0.174785100286533, 
    0.171919770773639, 0.169054441260745, 0.166189111747851, 
    0.163323782234957, 0.163323782234957, 0.160458452722063, 
    0.157593123209169, 0.154727793696275, 0.151862464183381, 
    0.148997134670487, 0.148997134670487, 0.146131805157593, 
    0.143266475644699, 0.140401146131805, 0.137535816618911, 
    0.134670487106017, 0.131805157593123, 0.128939828080229, 
    0.126074498567335, 0.123209169054441, 0.120343839541547, 
    0.120343839541547, 0.117478510028653, 0.114613180515759, 
    0.111747851002865, 0.108882521489971, 0.106017191977077, 
    0.103151862464183, 0.100286532951289, 0.0974212034383954, 
    0.0945558739255015, 0.0945558739255015, 0.0916905444126075, 
    0.0888252148997135, 0.0859598853868195, 0.0830945558739254, 
    0.0802292263610315, 0.0773638968481375, 0.0744985673352435, 
    0.0716332378223495, 0.0687679083094556, 0.0659025787965616, 
    0.0630372492836676, 0.0601719197707736, 0.0573065902578797, 
    0.0544412607449857, 0.0515759312320917, 0.0487106017191977, 
    0.0458452722063037, 0.0458452722063037, 0.0458452722063037, 
    0.0429799426934098, 0.0401146131805158, 0.0372492836676218, 
    0.0343839541547278, 0.0315186246418339, 0.0286532951289399, 
    0.0286532951289399, 0.0257879656160458, 0.0229226361031518, 
    0.0229226361031518, 0.0229226361031518, 0.0200573065902578, 
    0.0171919770773639, 0.0171919770773639, 0.0171919770773639, 
    0.0143266475644699, 0.0114613180515759, 0.0114613180515759, 
    0.00859598853868193, 0.00859598853868193, 0.00573065902578795, 
    0.00286532951289398, 0.00286532951289398, 0, 0), ss = c(1, 
    1.00286532951289, 1.00573065902579, 1.00859598853868, 1.01146131805158, 
    1.01432664756447, 1.01719197707736, 1.02005730659026, 1.02292263610315, 
    1.02578796561605, 1.02865329512894, 1.03151862464183, 1.03438395415473, 
    1.03724928366762, 1.00276652504693, 1.00563185455983, 1.00849718407272, 
    1.01136251358561, 1.01422784309851, 1.0170931726114, 1.0199585021243, 
    1.02282383163719, 1.02568916115008, 1.02855449066298, 1.03141982017587, 
    1.03428514968877, 1.03715047920166, 1.04001580871455, 1.04288113822745, 
    1.04574646774034, 1.04861179725324, 1.05147712676613, 1.05434245627902, 
    1.05720778579192, 1.06007311530481, 1.06293844481771, 1.0658037743306, 
    1.06866910384349, 1.07153443335639, 1.07439976286928, 1.07726509238218, 
    1.08013042189507, 1.08299575140796, 1.08586108092086, 1.08872641043375, 
    1.09159173994665, 1.05710898132596, 1.05997431083885, 1.06283964035174, 
    1.06570496986464, 1.06857029937753, 1.07143562889043, 1.07430095840332, 
    1.07716628791621, 1.08003161742911, 1.082896946942, 1.0857622764549, 
    1.08862760596779, 1.09149293548068, 1.09435826499358, 1.09722359450647, 
    1.10008892401937, 1.10295425353226, 1.10581958304515, 1.10868491255805, 
    1.11155024207094, 1.11441557158384, 1.11728090109673, 1.12014623060962, 
    1.12301156012252, 1.12587688963541, 1.12874221914831, 1.1316075486612, 
    1.13447287817409, 1.13733820768699, 1.14020353719988, 1.14306886671278, 
    1.14593419622567, 1.14879952573856, 1.15166485525146, 1.15453018476435, 
    1.15739551427725, 1.16026084379014, 1.16312617330303, 1.16599150281593, 
    1.16885683232882, 1.17172216184172, 1.17458749135461, 1.1774528208675, 
    1.1803181503804, 1.18318347989329, 1.18604880940619, 1.18891413891908, 
    1.19177946843197, 1.19464479794487, 1.19751012745776, 1.20037545697066, 
    1.20324078648355, 1.20610611599644, 1.20897144550934, 1.21183677502223, 
    1.21470210453512, 1.21756743404802, 1.22043276356091, 1.22329809307381, 
    1.2261634225867, 1.22902875209959, 1.23189408161249, 1.23475941112538, 
    1.23762474063828, 1.24049007015117, 1.24335539966406, 1.24622072917696, 
    1.24908605868985, 1.25195138820275, 1.25481671771564, 1.25768204722853, 
    1.26054737674143, 1.26341270625432, 1.26627803576722, 1.26914336528011, 
    1.272008694793, 1.2748740243059, 1.27773935381879, 1.28060468333169, 
    1.28347001284458, 1.28633534235747, 1.28920067187037, 1.29206600138326, 
    1.29493133089616, 1.29779666040905, 1.30066198992194, 1.30352731943484, 
    1.30639264894773, 1.30925797846063, 1.31212330797352, 1.31498863748641, 
    1.31785396699931, 1.3207192965122, 1.3235846260251, 1.32644995553799, 
    1.32931528505088, 1.33218061456378, 1.33504594407667, 1.33791127358957, 
    1.34077660310246, 1.34364193261535, 1.34650726212825, 1.34937259164114, 
    1.35223792115404, 1.35510325066693, 1.35796858017982, 1.36083390969272, 
    1.36369923920561, 1.36656456871851, 1.3694298982314, 1.37229522774429, 
    1.37516055725719, 1.37802588677008, 1.38089121628298, 1.38375654579587, 
    1.38662187530876, 1.38948720482166, 1.39235253433455, 1.39521786384745, 
    1.39808319336034, 1.40094852287323, 1.40381385238613, 1.40667918189902, 
    1.40954451141192, 1.41240984092481, 1.4152751704377, 1.38079241181701, 
    1.38365774132991, 1.3865230708428, 1.3893884003557, 1.39225372986859, 
    1.39511905938148, 1.39798438889438, 1.40084971840727, 1.36636695978658, 
    1.36923228929948, 1.37209761881237, 1.37496294832526, 1.37782827783816, 
    1.38069360735105, 1.34621084873036, 1.34907617824326, 1.35194150775615, 
    1.35480683726904, 1.35767216678194, 1.36053749629483, 1.36340282580773, 
    1.36626815532062, 1.36913348483351, 1.37199881434641, 1.3748641438593, 
    1.34038138523861, 1.34324671475151, 1.3461120442644, 1.34897737377729, 
    1.35184270329019, 1.35470803280308, 1.35757336231598, 1.36043869182887, 
    1.36330402134176, 1.36616935085466, 1.36903468036755, 1.37190000988045, 
    1.37476533939334, 1.37763066890623, 1.38049599841913, 1.34601323979844, 
    1.34887856931133, 1.31439581069064, 1.31726114020354, 1.32012646971643, 
    1.32299179922933, 1.32585712874222, 1.32872245825511, 1.33158778776801, 
    1.3344531172809, 1.3373184467938, 1.34018377630669, 1.34304910581958, 
    1.34591443533248, 1.34877976484537, 1.35164509435826, 1.35451042387116, 
    1.35737575338405, 1.36024108289695, 1.36310641240984, 1.36597174192273, 
    1.36883707143563, 1.37170240094852, 1.37456773046142, 1.37743305997431, 
    1.3802983894872, 1.3831637190001, 1.38602904851299, 1.38889437802589, 
    1.39175970753878, 1.39462503705167, 1.39749036656457, 1.40035569607746, 
    1.40322102559036, 1.40608635510325, 1.40895168461614, 1.41181701412904, 
    1.37733425550835, 1.38019958502124, 1.34571682640055, 1.34858215591345, 
    1.35144748542634, 1.31696472680565, 1.31983005631855, 1.32269538583144, 
    1.32556071534433, 1.29107795672364, 1.29394328623654, 1.29680861574943, 
    1.29967394526233, 1.30253927477522, 1.30540460428811, 1.30826993380101, 
    1.3111352633139, 1.3140005928268, 1.31686592233969, 1.31973125185258, 
    1.32259658136548, 1.32546191087837, 1.32832724039127, 1.33119256990416, 
    1.33405789941705, 1.33692322892995, 1.33978855844284, 1.30530579982215, 
    1.30817112933505, 1.31103645884794, 1.31390178836083, 1.31676711787373, 
    1.31963244738662, 1.32249777689952, 1.32536310641241, 1.3282284359253, 
    1.3310937654382, 1.29661100681751, 1.2994763363304, 1.3023416658433, 
    1.30520699535619, 1.30807232486908, 1.31093765438198, 1.31380298389487, 
    1.31666831340777, 1.31953364292066, 1.32239897243355, 1.32526430194645, 
    1.32812963145934, 1.33099496097224, 1.33386029048513, 1.33672561999802, 
    1.33959094951092, 1.34245627902381, 1.34532160853671, 1.3481869380496, 
    1.35105226756249, 1.35391759707539, 1.3194348384547, 1.32230016796759, 
    1.32516549748049, 1.32803082699338, 1.33089615650627, 1.33376148601917, 
    1.29927872739848, 1.30214405691137, 1.30500938642427, 1.30787471593716, 
    1.31074004545005, 1.31360537496295, 1.31647070447584, 1.31933603398874, 
    1.32220136350163, 1.32506669301452, 1.32793202252742, 1.29344926390673, 
    1.29631459341962, 1.29917992293252, 1.30204525244541, 1.3049105819583, 
    1.3077759114712, 1.31064124098409, 1.31350657049699, 1.31637190000988, 
    1.31923722952277, 1.28475447090208, 1.28761980041498, 1.29048512992787, 
    1.29335045944077, 1.29621578895366, 1.29908111846655, 1.30194644797945, 
    1.30481177749234, 1.30767710700524, 1.31054243651813, 1.31340776603102, 
    1.31627309554392, 1.31913842505681, 1.32200375456971, 1.3248690840826, 
    1.32773441359549, 1.33059974310839, 1.33346507262128, 1.29898231400059, 
    1.2644995553799, 1.2673648848928, 1.27023021440569, 1.27309554391859, 
    1.27596087343148, 1.27882620294437, 1.28169153245727, 1.24720877383658, 
    1.25007410334947, 1.25293943286237, 1.21845667424168, 1.18397391562099, 
    1.18683924513388, 1.18970457464677, 1.15522181602608, 1.12073905740539, 
    1.12360438691829, 1.12646971643118, 1.09198695781049, 1.09485228732339, 
    1.0603695287027, 1.06323485821559, 1.06610018772849, 1.0316174291078, 
    1.03448275862069, 1)), row.names = c(NA, -379L), class = c("roc_cutpointr", 
"tbl_df", "tbl", "data.frame"))

I tried run cutpointr without bootstraping and it seems that, in that case, manual and R results are the same. So there's the reason why this question was a bit useless, sorry for wasting your time and thanks a lot for your help!

Thie1e commented 4 years ago

OK, so you ran method = maximize_boot_metric before? Or did you now just set boot_runs = 0?

jgarces02 commented 4 years ago

No, I run in the "classical" way: cutpointr(data = mydata, x = myvar, class = myclass, na.rm = T)... running it with method = maximize_boot_metric would also solve this problem?

Thie1e commented 4 years ago

So you ran it either with boot_runs = 0 and with, for example, boot_runs = 500 and it returned different optimal cutpoints? If so, can you post mydata instead of the ROC curve? If it's very long, you can put it on pastebin.

jgarces02 commented 4 years ago

Yes, cutoffs for both tries are slightly different...

I didn't know pastebin very useful! Here you have mydata: https://pastebin.com/pBMu9zQR

Thie1e commented 4 years ago

Thanks for posting the data. I tried running cutpointr with and without bootstrapping. The results are (as they should be) the same. Also extracting the optimal value from the ROC curve returns the same cutpoint. Do you get differing cutpoints running this code? I used as.data.frame when extracting rows from the ROC curve to avoid the automatic rounding by tibble. Maybe that caused the discrepancy?

library(cutpointr)
library(tidyverse)

(cp1 <- cutpointr(dat, myvar, myclass, na.rm = T, metric = sum_sens_spec))
#> Assuming the positive class is 0
#> Assuming the positive class has higher x values
#> # A tibble: 1 x 16
#>   direction optimal_cutpoint method          sum_sens_spec      acc sensitivity
#>   <chr>                <dbl> <chr>                   <dbl>    <dbl>       <dbl>
#> 1 >=               0.0990444 maximize_metric       1.35766 0.671498    0.670157
#>   specificity      AUC pos_class neg_class prevalence outcome predictor
#>         <dbl>    <dbl> <fct>     <fct>          <dbl> <chr>   <chr>    
#> 1      0.6875 0.682714 0         1           0.922705 myclass myvar    
#>   data               roc_curve           boot 
#>   <list>             <list>              <lgl>
#> 1 <tibble [414 x 2]> <tibble [405 x 10]> NA
(cp2 <- cutpointr(dat, myvar, myclass, na.rm = T, metric = sum_sens_spec, boot_runs = 10))
#> Assuming the positive class is 0
#> Assuming the positive class has higher x values
#> Running bootstrap...
#> # A tibble: 1 x 16
#>   direction optimal_cutpoint method          sum_sens_spec      acc sensitivity
#>   <chr>                <dbl> <chr>                   <dbl>    <dbl>       <dbl>
#> 1 >=               0.0990444 maximize_metric       1.35766 0.671498    0.670157
#>   specificity      AUC pos_class neg_class prevalence outcome predictor
#>         <dbl>    <dbl> <fct>     <fct>          <dbl> <chr>   <chr>    
#> 1      0.6875 0.682714 0         1           0.922705 myclass myvar    
#>   data               roc_curve           boot              
#>   <list>             <list>              <list>            
#> 1 <tibble [414 x 2]> <tibble [405 x 10]> <tibble [10 x 23]>

cp1 %>% 
  select(roc_curve) %>% 
  unnest(cols = "roc_curve") %>% 
  slice(which.max(m)) %>%
  as.data.frame
#>     x.sorted  tp fp tn  fn       tpr    tnr    fpr       fnr        m
#> 1 0.09904436 256 10 22 126 0.6701571 0.6875 0.3125 0.3298429 1.357657

cp2 %>% 
  select(roc_curve) %>% 
  unnest(cols = "roc_curve") %>% 
  slice(which.max(m)) %>%
  as.data.frame
#>     x.sorted  tp fp tn  fn       tpr    tnr    fpr       fnr        m
#> 1 0.09904436 256 10 22 126 0.6701571 0.6875 0.3125 0.3298429 1.357657

Created on 2020-07-13 by the reprex package (v0.3.0)

jgarces02 commented 4 years ago

That's a mystery... I ran this code:

> ct0 <- cutpointr(data = mydata, x = myvar, class = myclass, na.rm = T, method = maximize_boot_metric, boot_stratify = T, boot_runs = 0)
> ct0
# A tibble: 1 x 16
  direction optimal_cutpoint method               sum_sens_spec      acc sensitivity specificity      AUC pos_class neg_class
  <chr>                <dbl> <chr>                        <dbl>    <dbl>       <dbl>       <dbl>    <dbl> <fct>     <fct>    
1 >=               0.0820404 maximize_boot_metric       1.31888 0.714976    0.725131     0.59375 0.682714 0         1        
  prevalence outcome predictor data               roc_curve          boot 
       <dbl> <chr>   <chr>     <list>             <list>             <lgl>
1   0.922705 myclass myvar     <tibble [414 x 2]> <tibble [405 x 9]> NA   

> ct500 <- cutpointr(data = mydata, x = myvar, class = myclass, na.rm = T, method = maximize_boot_metric, boot_stratify = T, boot_runs = 500)
> ct500
# A tibble: 1 x 16
  direction optimal_cutpoint method               sum_sens_spec      acc sensitivity specificity      AUC pos_class neg_class
  <chr>                <dbl> <chr>                        <dbl>    <dbl>       <dbl>       <dbl>    <dbl> <fct>     <fct>    
1 >=               0.0823939 maximize_boot_metric       1.31888 0.714976    0.725131     0.59375 0.682714 0         1        
  prevalence outcome predictor data               roc_curve          boot               
       <dbl> <chr>   <chr>     <list>             <list>             <list>             
1   0.922705 myclass myvar     <tibble [414 x 2]> <tibble [405 x 9]> <tibble [500 x 23]>

And these are different :cry:

Thie1e commented 4 years ago

Well, here you're using maximize_boot_metric, which bootstraps the cutpoint estimation, then you're running an outer bootstrapping which estimates the metrics after bootstrapping the optimal cutpoint (see the "bootstrapped cutpoints" chapter in the vignette for details).

If you set a seed before each cutpointr call using set.seed(123), the optimal cutpoints should be identical. They might of course differ from the optimal cutpoint extracted from the ROC curve.

jgarces02 commented 4 years ago

Perfect. Yes, indeed, with set.seed() all seems to work nicely. Thanks a lot for your help (and patience)!

Thie1e commented 4 years ago

You're welcome, that's good to hear. It's always helpful to see what users are struggling with.