DeLong AUC Confidence Interval

[anaphylacticapps]

I think there may be an issue with the DeLong confidence interval for AUC. When the sample size gets large, the CI goes to either 0-0 or 1-1. Here is an example:

Create ROC Objects

predictor1 <- c(runif(12000,0,0.5), runif(14472-12000, 0.5,0.75)) response1 <- rbinom(14472, size=1, p=predictor1) roc1 <- roc(response1, predictor1)

predictor2 <- c(runif(3 12000,0,0.5), runif(3 (14472-12000), 0.5,0.75)) response2 <- rbinom(3 * 14472, size=1, p=predictor2) roc2 <- roc(response2, predictor2)

predictor3 <- c(runif(10 12000,0,0.5), runif(10 (14472-12000), 0.5,0.75)) response3 <- rbinom(10 * 14472, size=1, p=predictor3) roc3 <- roc(response3, predictor3)

Calculate AUC and CI

auc(roc1) Area under the curve: 0.7586 ci.auc(roc1) 95% CI: 0.7506-0.7667 (DeLong)

auc(roc2) Area under the curve: 0.7584 ci.auc(roc2) 95% CI: 0.7537-0.7631 (DeLong)

auc(roc3) Area under the curve: 0.7561 ci.auc(roc3) 95% CI: 1-1 (DeLong)

[xrobin]

@sieste could you please have a look at this? It looks like the new DeLong code is returning erroneous theta values, maybe an overflow or something like that?

> pROC:::delongPlacementsCpp(roc3)$theta
[1] 15.97072

@anaphylacticapps as workaround, the old version 1.8 of pROC reports correct results, although it takes significantly more time to do so:

> ci.auc(roc3)
95% CI: 0.7567-0.7618 (DeLong)
> pROC:::delongPlacementsCpp(roc3)$theta
[1] 0.7592556

[sieste]

That should do the trick, sorry about that. I didn't expect someone would need a confidence interval at a sample size so big that it causes integer overflow ;)

Cheers, Stefan

[xrobin]

Excellent, thanks a lot for the very quick fix!

This is the same as commit e71e57b then. I'll add a check on the R side to make sure theta == auc to catch similar cases in the future. Then push a bugfix release asap.

[anaphylacticapps]

Thank you, both.

[xrobin]

There is now an error message when the DeLong AUC doesn't match.

Thanks @anaphylacticapps for the report!