Treat a study that compares a single reader against multiple readers

DIDSR / iMRMC

iMRMC: Software to do multi-reader multi-case analysis of reader studies

http://didsr.github.io/iMRMC/

Other

22 stars 17 forks source link

Treat a study that compares a single reader against multiple readers #5

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago

Please describe the desired enhancement

I am a student from ****** ******. I downloaded your iMRMC software and would 
like to use it to analyze a dataset. My dataset includes 10 readers, 1 modality 
and the ratings for about 100 cases. The first reader is CAD and the rest nine 
readers are radiologists. My study is about the comparison between CAD and the 
average performance of all radiologists. 

Could you please tell me if it is possible to perform such kind of analysis 
using the iMRMC software?

Thank you very much!

Best regards,

****** ******

Original issue reported on code.google.com by Brandon.Gallas on 10 Feb 2014 at 3:20

GoogleCodeExporter commented 9 years ago

Original comment by Brandon.Gallas on 27 Feb 2014 at 8:24

Changed state: AssignTo_BDG

marta-pinto commented 4 years ago

Hi! I downloaded the iMRMC 4.0.3 software and I am performing the same type of analysis between a CAD system and radiologist readers. The problem is that for the readers I have a scale of 1-100 and for the CAD system I have a scale from 1-10 (with decimal values as scores, such as 0.41, 1,23, etc.). Therefore, I end up with a ROC curve for readers with multiple points but for the CAD system I have only 10 points in the ROC curve, even though I am using decimal values that should give me a bigger set of point in the curve. Am I doing something wrong? Thank you very much for your attention.

brandon-gallas commented 4 years ago

iMRMC analyzes numeric data. It will see two numbers as distinct if they are different, no matter the decimal place, 5.01 > 5.001. You may get 10 points (not counting horizontal and vertical segments) on the ROC curve if your data bunches into groups when sorted by score (10 signal-present in a row, 15 signal-absent in a row, etc.). It's hard for me to say more without seeing the data. Try plotting the data. Let x be the score and y be 1 for signal-present and 2 for signal absent.

marta-pinto commented 4 years ago

I am attaching a picture with the two ROC curves I get (one for a reader and another for the CAD system) and also some examples of the scores I used. Could you see if this helps to know why I get approximately 10 points in the AI ROC curve and more points in the readers ROC curve? Thank you very much for the fast answer!

brandon-gallas commented 4 years ago

This does not help. Try plotting the data as symbols. Let x be the scores and y be the truth. I've attached a sample figure below.

marta-pinto commented 4 years ago

So, like this?

brandon-gallas commented 4 years ago

OK. It's not entirely clear, but it looks like your CAD algorithm yields about 19 scores for signal-present cases (or there are only 19 signal-present cases). It looks like your algorithm probably outputs a lot of ties. In your previous image, you can see that 10.0 appears twice and 9.99 appears twice. There must be a lot of those because the ROC curve jumps from (0,0) to approximately (0.13, 0.81). There are no operating points in between because there are no signal-absent scores in between. I don't think you are doing anything wrong. Your data has ties.

marta-pinto commented 4 years ago

Ok, thank you for helping me understand! I will check how I can change the CAD algorithm part.