brandon-gallas
commented
1 week ago Hi Craig,
Significance for your small difference in AUCs must be coming from a high level of correlation across modalities. Remember, var(A-B) = var(A) + var(B) - 2cov(A,B). If cov(A,B) is close to var(A) and var(B), your precision on var(A-B) can be very good.
BTW, the normal approximation is expected to show "more significance" than the t-distribution because it assumes the variance is known exactly.
The variance components are an output of the software. Perhaps you are using the java gui. This is a static piece of software that is no longer being maintained. I recommend that you use the R package moving forward. You can find information here: iMRMC: Software to do Multi-reader Multi-case Statistical Analysis of Reader Studies | Center for Devices and Radiological Health (fda.gov). Here is the R package reference manual, which documents all the functions: https://cran.r-project.org/web/packages/iMRMC/iMRMC.pdf. Page 8 shows the output of the doIMRMC function. The output you are looking for is varDecomp which contains components of variance and the corresponding coefficients. The components are described in this paper:
- B. D. Gallas, A. Bandos, F. Samuelson, and R. F. Wagner, “A framework for random-effects ROC analysis: biases with the bootstrap and other variance estimators,” Commun Stat A-Theory, vol. 38, no. 15, pp. 2586–2603, 2009, doi: 10.1080/03610920802610084.
I believe that the java gui also provides the components of variance somewhere in the interface.
ckabbey
This is an edited version of my original post. It is more focused on the issue I could use some help with.
Hello Brandon or whoever is reading this,
I am working with some colleagues who are using the iMRMC software to analyze fully-crossed MRMC data with more than 100 cases (half positive) and more than 10 readers. In the iMRMC results, the average AUC difference between the treatment and control conditions is less than 0.015, so pretty small. The iMRMC analysis with df(BDG) finds this difference to be significant with a p-value less than 0.01, and it's even smaller under the normal approximation.
This seems like high significance for such a small difference, and I am trying to understand how this came to be. We have reason to believe that readers should be relatively consistent between the two conditions, but it would be great to be able to link that qualitative understanding to some quantitative component of the test statistic (e.g. the treatment-by-reader variance component in the DBM model). Is there some way to decompose the iMRMC test statistic into these types of components as a way to understand our finding.
If this has been developed in one of your papers, please point us to the reference.
Any help appreciated!
Sincerely, Craig Abbey