search

DIDSR / iMRMC

iMRMC: Software to do multi-reader multi-case analysis of reader studies
http://didsr.github.io/iMRMC/
Other
22 stars 17 forks source link

Not Fully Crossed Analysis (Binary Data) #153

Closed stevenhurwitt closed 4 years ago

stevenhurwitt commented 5 years ago

Is there any way to manipulate the code to work with a non-fully crossed design? For instance if readers are divided up across the cases (and every reader does not read every case), then the AUC values are only a fraction of what they should be.

Is there any literature on how to adjust for this, or any functionality within the current code?

brandon-gallas commented 5 years ago

You are in luck. The code can do it. That's why we have a list-mode primary data format.

Gallas, B. D. & Brown, D. G. (2008), 'Reader Studies for Validation of CAD Systems', Neural Networks Special Conference Issue 21(2), 387-397.

Gallas, B. D.; Pennello, G. A. & Myers, K. J. (2007), 'Multireader Multicase Variance Analysis for Binary Data', J Opt Soc Am A, Special Issue on Image Quality 24(12), B70-B80.

stevenhurwitt commented 5 years ago

Thanks! I will look into it in the next few days - the naive way I tried was giving AUC's of around .5 and I assume averaging over all readers (they should have been around .9 with the sample data I was using)