search

greenelab / mpmp

Multimodal Pan-cancer Mutation Prediction
BSD 3-Clause "New" or "Revised" License
7 stars 6 forks source link

Run plot scripts with AUROC instead of AUPR #57

Closed jjc2718 closed 3 years ago

jjc2718 commented 3 years ago

One of our collaborators suggested that it might be good to generate results with AUROC in addition to AUPR (since this is a common thing for reviewers to ask about). I think we'll probably convert the figure generation notebook generate_figures_auroc.ipynb to a PDF and include it or link to it as a supplementary artifact with our paper, just to pre-empt questions about whether the metric matters or not.

Qualitatively all of the results with AUROC are essentially the same, changing classification metrics doesn't seem to matter much.

I implemented this by adding a flag to the top of each script; let me know if you have other ideas. This is a bit similar to some of the changes in #56 and I could take a similar approach, but I'm not sure it makes sense to have a separate version of each script for AUROC. Alternatively, I could have a single flag (e.g. in mpmp/config.py) and have this flag control AUPR vs. AUROC in all the scripts, so it only has to be changed in one spot - curious what you think is best.

jjc2718 commented 3 years ago

I think 11 days may be a new record amount of time it has taken me to review a PR.

No worries! I assigned it the Friday before retreat week so I definitely didn't expect you to take a look right away 😂

I don't know if I've asked this before, but is there a reason that you use the SAVE_FIGS flag instead of just overwriting the existing figures? Does it take awhile to save everything?

It does take a bit longer, and also sometimes I don't want to overwrite the existing versions of the figures (e.g. if I fixed a bug or generated some new data, and I want to check that the results makes sense before I remake the paper figures). I could just change all the filenames or comment the code out each time, but it seemed like this case was coming up often enough that it was worth creating a flag for it.