search

DIDSR / iMRMC

iMRMC: Software to do multi-reader multi-case analysis of reader studies
http://didsr.github.io/iMRMC/
Other
22 stars 17 forks source link

Problem about 1 distribution ROC curve #42

Closed qigongFDA closed 9 years ago

qigongFDA commented 9 years ago

Hello Brandon,

When I do the not fully crossed ROC curve task, I find a problem. I build an input file. A reader only reads one case for one modality, and reads all other modalities cases. The java gives me error. I think the error is coming from that reader and modality, there is only one PDF (because the case that reader reads should either normal or disease.). Then we can't get one of TPF and FPF, and can't get ROC.

I ask Frank and Weijie about this. They answer is, we can't get ROC curve if truth of cases are all positive or negative.

I could add one check function in imrmc to detele all the data for that reader & modality. Or do you have any idea to make the ROC curve. Thank you

brandon-gallas commented 9 years ago

Hi Q,

You have generally identified the problem and the fix. When describing the fix, please identify where in the code you expect to make changes. I think changes should happen in InputFile.verifySizesAndGetIDs. If a reader does not read at least 2 normal cases and 2 disease cases for a modality, delete the scores for that reader and modality. Also alert the software user to the fact that iMRMC is “ignoring” those scores. This requires an input file to test. I have (re)created a folder that held input files for debugging. You can find it at https://github.com/DIDSR/iMRMC/tree/gh-pages/000_iMRMC/inputFiles/forDebugging Please put useful input files in this directory. Try and name the file to describe its purpose, and then add more details in the free-text portion of the input file.

Thanks for creating this issue and finding a solution!

Brandon

qigongFDA commented 9 years ago

Hello Brandon,

OK. I just added a new input file to that folder.

When I asked Frank, he said you once mentioned some method. It could help us calculate AUC even all the data have same truth. I thought you might also have method to plot ROC curve under that situation. Thus, I just mentioned another alternative method briefly, and hadn't looked the code and found a proper place to put the function. I will try to describe them more detail in future. Thank you.

Sincerely, Qi

brandon-gallas commented 9 years ago

I don’t understand most of your comment. Let’s talk about this during our 1on1, Tuesday.

From: Qi Gong [mailto:notifications@github.com] Sent: Friday, October 02, 2015 4:29 PM To: DIDSR/iMRMC Cc: Gallas, Brandon D. Subject: Re: [iMRMC] Problem about 1 distribution ROC curve (#42)

Hello Brandon,

OK. I just added a new input file to that folder.

When I asked Frank, he said you once mentioned some method. It could help us calculate AUC even all the data have same truth. I thought you might also have method to plot ROC curve under that situation. Thus, I just mentioned another alternative method briefly, and hadn't looked the code and found a proper place to put the function. I will try to describe them more detail in future. Thank you.

Sincerely, Qi

— Reply to this email directly or view it on GitHubhttps://github.com/DIDSR/iMRMC/issues/42#issuecomment-145146374.

brandon-gallas commented 9 years ago

Got it. Thanks. Done.