DIDSR / iMRMC

iMRMC: Software to do multi-reader multi-case analysis of reader studies
http://didsr.github.io/iMRMC/
Other
22 stars 17 forks source link

Let Show ROC Curve button works for not fully crossed data #41

Closed qigongFDA closed 8 years ago

qigongFDA commented 8 years ago

If the the input file is not fully crossed. (i.e. Some modalities not read by all readers), the ROC button produce error. Solve this problem to let the software more robust.

qigongFDA commented 8 years ago

In 846 version of qigong branch, ROC curve could plot for not fully crossed input data.

qigongFDA commented 8 years ago

After add code for issue 42, ROC curve has a little problem. The input file has same data for modality 1,3 and 2,4. But now ROC curve is not exactly overlap for this two groups.

qigongFDA commented 8 years ago

In generateROCpoints function, I find the keyedData is loaded correctly.

And I find out the reason for this problem might be the variable max and min ( about 320 lines in InputFile.java). Our code generate ROC data modality by modality. For each modality, we calculate minimum and maximum score over all the readers, and set 100 threshold steps inside them. As the maximum and minimum is over all readers, but not each reader, the ROC curve might not overlap exactly.

The following is what we did for debugging purposes. We assumed we had 3 readers and 1 modality A. Reader 2 gave the highest and lowest scores. We artificially duplicated the data and named it as modality B. Then we deleted all the data about reader 2 in modality B to see how the ROC plotting procedures reacted. When we processed modality A, the maximum and minimum for ROC is reader 2's scores. But when we processed modality B, the maximum and minimum are the highest and lowest scores from reader 1 and 3. Now even though two modalities have same data for reader 1 and 3, the threshold check points for the ROC curves are different across modalities. They don't overlap exactly. @brandon-gallas

brandon-gallas commented 8 years ago

Perfect