search

lindawangg / COVID-Net

COVID-Net Open Source Initiative
Other
1.15k stars 480 forks source link

Why no validation data? #66

Open axkoenig opened 4 years ago

axkoenig commented 4 years ago

Hello, I have a general question regarding the COVIDx dataset. You define which images to use as train and test data in the files train_COVIDx3.txt and test_COVIDx3.txt. However, why don't you also define and use a validation set? In your paper you describe that the architecture was determined using "Generative Synthesis" - a machine-driven design exploration. On which dataset exactly did you compare the performance of different architectures and hyper-paramaters that the Generative Synthesis approach produced? It is good practice to do this on a validation set, right? Looking forward to any comments and implementation details on this :) Alex

sergevanhaag2 commented 4 years ago

I agree. I am not able to recreate the results if I use a training, validation, and test-set. I am afraid that COVID-NET indirectly overfits on the test-set. I'd suggest using the test-set only once.

sergevanhaag2 commented 4 years ago

@lindawangg Please look into this. I do not report the same accuracy when I make use of different test-sets. I would suggest doing a more thorough examination of the results. I do think the reported results are too optimistic at the moment. Rather than repeatedly creating new datasets, why is the focus not on improving this?