Closed nottombrown closed 5 years ago
Cleverhans now has a canonical implementation checked in.
Using the new spatial attack, I get the following results on the keras resnet
+------------------------------+----------------------+----------------------+
| Attack name | Acc @ 80% cov | Acc @ 100% cov |
+------------------------------+----------------------+----------------------+
| clean | 1.0 | 1.0 |
| spatial_grid | 0.965 | 0.904 |
| spsa_with_random_spatial | 0.016249999999999987 | 0.013000000000000012 |
| boundary_with_random_spatial | 0.040000000000000036 | 0.03125 |
+------------------------------+----------------------+----------------------+
Solved by #50
I am trying out the updated attack. It is taking 35 s/iteration. Is this to be expected? Also, my Google Colab instance (with GPU) is dying after 4%. Usually that happens when some memory limit is reached. Does running the _spatialgrid attack take lot of memory?
35s/iter seems correct to me. I can profile it to see if we can speed it up.
Interesting. I can profile the memory to take a look.
It would be great if you could profile. I tried running it multiple times on Colab, but runtime is always running at 4%.
@sibyjackgrove - Ok take three! You can try out https://github.com/google/unrestricted-adversarial-examples/pull/52 if you'd like.
I'm not sure how many CPUs a colab env has. If it's too slow, I would recommend spinning up a GCE instance.
Will you be commiting this #52 to master soon? Colab has 4 CPU's.
Yes, I plan to commit this to master after battle-testing it a bit more On Fri, Oct 5, 2018 at 8:38 AM Siby Jose Plathottam < notifications@github.com> wrote:
Will you be commiting this #52 https://github.com/google/unrestricted-adversarial-examples/pull/52 to master soon? Colab has 4 CPU's.
— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/google/unrestricted-adversarial-examples/issues/46#issuecomment-427408365, or mute the thread https://github.com/notifications/unsubscribe-auth/AASt32bDNw4lSHspyXC5XGxVaDl58nVCks5uh3z-gaJpZM4W5Wji .
Closed by #52
I tried out the eval_kit
with the new spatial_grid attack on Google Colab. It is significantly faster compared to earlier attack and I am able to complete it in 1.5 hours despite the Colab VM only having 4 CPU's.
Good to hear!
This attack should take no more than 1 hour.
I ran it for 24 hours on the
undefended_keras_resnet
and it only completed 50%.This replaces the attack with a faster implementation that is pending in cleverhans. https://github.com/tensorflow/cleverhans/pull/623
I've copied the attack into our repo for now, and will swap to the cleverhans version after it is released.