search

samet-akcay / ganomaly

GANomaly: Semi-Supervised Anomaly Detection via Adversarial Training
MIT License
853 stars 213 forks source link

Is the testing function missing self.netg.eval? #83

Open diviramon opened 3 years ago

diviramon commented 3 years ago

Hi,

Sorry for opening another issue, but it seems like the code has an error. I was wondering why during test, self.netg.eval() is never called. This would lead to data leakage since the batchnorms would adjust statistics to the testing dataset. I know that the test is ran with torch.noGrad but that does not prevent it from updating the batchnorm layers,

I checked the tensorflow implementation and it is called there, but it does not happen in the pyTorch one.

Thanks!

JasperYangGithub commented 3 years ago

Thanks for coming up with this problem. It helped me a lot to find out the bug. self.netg.eval() should be added for model evaluation to disable BN and dropout.

tobybreckon commented 3 years ago

This has come up before with the associated skip-ganomaly repo (https://github.com/samet-akcay/skip-ganomaly)

I asked Samet Akcay, the first author [I am also a co-author on the GANomaly paper but my familiarity with the code-base is essentially zero] and he said "..it’s not a bug. During the DCGAN era, it was common not to put the model on eval mode for the stability. What I did instead, is to use batch norm, and batch size of 1 during the test so the performance is not affected."

If you add .eval() it significantly reduces the AUC performance against this batch size of 1 approach - ourselves and others have found this. AFIAK, there is no dropout in use in this approach.

Hope this helps.

YangjiaqiDig commented 2 years ago

Thank you! this issue almost killed me, after adding the eval(). outputs are the same for different number of batches now..

caiyu6666 commented 1 year ago

Thank you very much for the answser!

However, I am not sure about the reason why the train mode can improve the stability of DCGAN. Additionally, why does the eval mode reduce the performance?

Could you please help explain these questions?

Thanks again!

caiyu6666 commented 1 year ago

"During the DCGAN era, it was common not to put the model on eval mode for the stability. What I did instead, is to use batch norm, and batch size of 1 during the test so the performance is not affected."

However, I found that if we remove the adversarial training (i.e., $L_{adv}$ and the discriminator) and remain the second encoder, putting the model on train mode can also achieve considerably better performance. In contrast, if we use vanilla AE without extra encoder, the train mode cannot improve the performance.

Therefore, I think this operation, adding .eval() or not, has nothing to do with the DCGAN. Could you please explain it?