Closed sibyjackgrove closed 5 years ago
Yes, clean accuracy is the accuracy on the un-modified test set. The results of the other attacks will show below the clean results, in the same table.
Yes, clean accuracy is the accuracy on the un-modified test set. The results of the other attacks will show below the clean results, in the same table.
That doesn't seem to be the case when I use it. I see accuracy for only 'clean'. Please see the screenshot below:
Sorry, this was my mistake. I pushed a change that disabled the full suite of attacks. If you update to HEAD
and re-run your command, you should see the complete grid rather than only the clean
attack.
Thanks, I updated and I am re-running eval_kit.evaluate_bird_or_bicycle_model(my_custom_CNN_model)
. Here is what I am getting. The _spatial_grid_ attack seems to be very slow taking 128 s/iteration. Is this normal or is it my model's problem. The model I am using has 1.1 M parameters.
Now it is taking 372 s/iteration. I am using the Google Colab with GPU accelerator. Please let me know if is to be expected.
Yeah, unfortunately we need to improve that, see #46
Closing this issue because it is known. Perhaps we should puta larger warning that the warm-up evaluation is very slow?
Yeah, unfortunately we need to improve that, see #46
Is there any update on this? How can I use the attack reference in #46 ?
@sibyjackgrove - How about we discuss this in issue #46?
I followed the instructions for the warm-up attacks and used the API as shown below.
Here is part of my output:![image](https://user-images.githubusercontent.com/25213730/46035067-c7eca600-c0c7-11e8-8482-085fb212ef6d.png)
However, I am confused by the outputs. What does 'clean' attack mean? Is this the accuracy on a normal dataset (without adversarial inputs)? Is there another method to evaluate the model on the adversarial images?