SpatialAttack in eval_kit is sometimes extremely slow

[nottombrown]

This attack should take no more than 1 hour.

I ran it for 24 hours on the undefended_keras_resnet and it only completed 50%.

This replaces the attack with a faster implementation that is pending in cleverhans. https://github.com/tensorflow/cleverhans/pull/623

I've copied the attack into our repo for now, and will swap to the cleverhans version after it is released.

[nottombrown]

Cleverhans now has a canonical implementation checked in.

[nottombrown]

Using the new spatial attack, I get the following results on the keras resnet

+------------------------------+----------------------+----------------------+
| Attack name                  | Acc @ 80% cov        | Acc @ 100% cov       |
+------------------------------+----------------------+----------------------+
| clean                        | 1.0                  | 1.0                  |
| spatial_grid                 | 0.965                | 0.904                |
| spsa_with_random_spatial     | 0.016249999999999987 | 0.013000000000000012 |
| boundary_with_random_spatial | 0.040000000000000036 | 0.03125              |
+------------------------------+----------------------+----------------------+

[nottombrown]

Solved by #50

[sibyjackgrove]

I am trying out the updated attack. It is taking 35 s/iteration. Is this to be expected? Also, my Google Colab instance (with GPU) is dying after 4%. Usually that happens when some memory limit is reached. Does running the _spatialgrid attack take lot of memory?

[nottombrown]

35s/iter seems correct to me. I can profile it to see if we can speed it up.

Interesting. I can profile the memory to take a look.

[sibyjackgrove]

It would be great if you could profile. I tried running it multiple times on Colab, but runtime is always running at 4%.

[nottombrown]

@sibyjackgrove - Ok take three! You can try out https://github.com/google/unrestricted-adversarial-examples/pull/52 if you'd like.

I'm not sure how many CPUs a colab env has. If it's too slow, I would recommend spinning up a GCE instance.

[sibyjackgrove]

Will you be commiting this #52 to master soon? Colab has 4 CPU's.

[nottombrown]

Yes, I plan to commit this to master after battle-testing it a bit more On Fri, Oct 5, 2018 at 8:38 AM Siby Jose Plathottam < notifications@github.com> wrote:

Will you be commiting this #52 https://github.com/google/unrestricted-adversarial-examples/pull/52 to master soon? Colab has 4 CPU's.

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/google/unrestricted-adversarial-examples/issues/46#issuecomment-427408365, or mute the thread https://github.com/notifications/unsubscribe-auth/AASt32bDNw4lSHspyXC5XGxVaDl58nVCks5uh3z-gaJpZM4W5Wji .

[nottombrown]

Closed by #52

[sibyjackgrove]

I tried out the eval_kit with the new spatial_grid attack on Google Colab. It is significantly faster compared to earlier attack and I am able to complete it in 1.5 hours despite the Colab VM only having 4 CPU's.

[nottombrown]

Good to hear!