brandon-gallas
commented
1 year ago Jasper, You caught me at a busy time: grant proposal and regulatory assignments and modern life. I am actively researching agreement methods. My colleague and I have published (this paper) and a follow up for arbitrary study designs. I'm not sure ROC scores totally fit that work since they are not calibrated quantitative measurements (ratio data ... with units ... probabilities are units). I don't have anything ready to go. Here is a recent presentation I haven't posted yet to give you a flavor of where I am headed. Gallas2022_JSM-final20220808.pdf
- The slides show me thresholding the test and the reference standard to get sensitivity and specificity.
- You can, instead, threshold just the reference standard and calculate AUC. This might be more appropriate for ROC scores.
- If you then average over all thresholds, you get a pairwise concordance metric. See this paper.
I assume you wanted an answer three weeks ago, but if you have some more time to wait, more will be coming. Please check in again.
BTW, the FDA reviewed and recognized my tool in a more formal way. Check it out: https://www.fda.gov/medical-devices/science-and-research-medical-devices/imrmc-software-statistical-analysis-multi-reader-multi-case-reader-studies
Dear Brandon Gallas, dear all,
For the same study as mentioned in this issue Sizing reader study (radiologists versus standalone CAD) #168, we want to investigate several secondary objectives that specifically relate to the reader study we conduct.
To summarize our reader study, we have 50+ readers divided among four split plots to detect clinically significant prostate cancer lesions using two image configurations / modalities. Readers assess their allotted cases firstly under the first modality (bi-parametric prostate MRI), annotating and scoring lesions (available scores 3,4,5) and overall patient level suspicion (0-100 score). In a second step, new information is provided and readers reassess the same case using the second modality (multi-parametric prostate MRI). Our main goal was to compare these assessments with standalone AI readings, but we are also interested in comparing the two reading configurations.
For AUC analysis (and also for sens/spec) we are familiarizing ourselves with the JAVA GUI of iMRMC. In addition, we want to investigate the inter-reader agreement in scoring (namely the index lesion = highest ranking lesion score within a case). We found various papers by you and your team that investigate agreement analysis within MRMC (e.g. Chen et al., 2014, Gallas et al., 2016), but are not sure whether agreement analysis as we want to investigate within our study applies to these publications, and secondly, can be done with the iMRMC software.
We wanted to check if there is a possibility to trick the software, such as can be done with the analysis of binary data (as mentioned here: MRMC analysis of binary data) to perform this analysis, or that you have other ideas how to implement this with either the JAVA GUI or R package.
Thanks for providing this software and the many publications related to MRMC. They are of great help.
Best regards,
Jasper