search

arpanmangal / CovidAID

COVID-19 Detection Using Chest X-Ray
http://arxiv.org/abs/2004.09803
61 stars 38 forks source link

Replicating paper's results #8

Closed bartmch closed 4 years ago

bartmch commented 4 years ago

Hi, I was just trying to replicate the ROC's you presented in your paper using the test_set you described in prepare_data.py. I'm executing trainer.py with your proposed settings:

                "--mode","test",
                "--checkpoint","path_to_CovidAID_transfered.pth.tar",
                "--cm_path","path_to_plots/cm_best",
                "--roc_path","path_to_plots/roc_best",
                "--combine_pneumonia"

I see that default batch size is 64 which brings the total inputs.shape to [640,3,224,224] (default 10 crops). This doesn't fit on GPU's memory. I was wondering what other settings you used to validate on the test set? E.g.: did you added a specific batch size "--bs","1" or did you use the distributed processing - if so can you explain to me how to use this in your code? Thanks!

arpanmangal commented 4 years ago

batch_size at test time does not matter. You can use any size which fits in your GPU memory. For training, see sec. 3.3 of our paper for hyper-parameters.

bartmch commented 4 years ago

I just lowered the batch size and it runs fine (testing bs shouldn't matter - you're right). A more urgent question: I created the ROCs and CMs for the data split in the code (which reflects the paper). When I want to replicate the results using: ["--mode","test","--freeze","--checkpoint","path_to/CovidAID_transfered.pth.tar","--bs","8","--cm_path","path_to/plots/cm_best","--roc_path","path_to/plots/roc_best","--combine_pneumonia"] I get the following ROC and CM: image image image

arpanmangal commented 4 years ago

Which pre-trained model did you use? Did you use the correct model?

bartmch commented 4 years ago

Which pre-trained model did you use? Did you use the correct model?

I followed your steps. The only difference is that I had to make the model torch>1.0 compatible but I didn't change any weights or so, only the names of the layers - that shouldn't really matter right? FYI: test.txt

bartmch commented 4 years ago

Apologies - I just figured out I was pointing to the wrong trained network! Everything looks good now. image