Closed carlini closed 5 years ago
Dear Nicholas,
Thanks so much for trying our defense method.
We tested your code just now and found that once 'model.eval()' is added in the code, the issue will be fixed. This is simply because WideResNet has the batch-normalization operator, and model.eval() will help to fix the batch normalization during the test time. Please let us know if you have further question. Thanks again.
Best, Hongyang
Ah, well this is where my lack of PyTorch knowledge really shows. Thanks.
Thank you very much for releasing the model and associated code along with your paper. I'm very grateful that you've put the effort into making it as easy as possible to get everything up and running, and I sincerely hope others involved in the contest follow your lead.
I'm taking an initial first pass at looking at everything, and am getting somewhat confusing results. First, it looks like the model is giving very different results at small batch sizes:
will output
It looks like there is also some dependence on the data of the batch to classify each input. I have some batch of 10000 examples I want to process as a [100, 100, 3, 32, 32] matrix, and if I process them in row-major order I get a different accuracy than column-major. I suspect this might have the same underlying cause, so I'll give details for that later if necessary.
As you might imagine, this makes it difficult to evaluate the defense: evaluating the network with a batch of [99 clean examples] + [1 adversarial example] gives a different result than [50 clean examples] + [50 adversarial examples].
Is this intended, am I doing something wrong, or something else?