More efficient projections for FAB

[jeromerony]

Hi!

As the title says, here are slightly more efficient projections used in FAB. The main differences for the implementation are:

• it does not use nonzero();
• it does not move tensors from cpu to device (instead creates them directly on device);
• it avoids unnecessary cloning.

Cheers

[fra31]

Hi,

thanks for the contribution! I'll try to quickly have a look and merge it. Do you roughly know which is the improvement in terms of time?

[jeromerony]

Sorry, I should have included them in the first place.

I ran a few benchmarks on a Titan RTX. First a synthetic benchmark for the projections alone (with cudnn.benchmark enabled). The results are the following (time in seconds) given the same data:	Projection	Original	New
L1	1.79	0.522
L2	0.565	0.35
Linf	0.756	0.338

So the improvements depend on the projections.

In term of run-time for FAB, I tried with the eval script (ResNet-18, 1000 samples of CIFAR10, Linf at 1/255) and, with the new projections, the attack is about 2% faster (317.6s instead of 323.6s) with cudnn.benchmark enabled. Obviously this difference is going to be smaller for larger models and vice versa.

[fra31]

Ok, thanks a lot, that's a significant improvement.

Yeah, for the full attack anyway the other computations should dominate in terms of time. But it makes definitely sense to have more efficient projections in general.

[jeromerony]

I made some other minor optimizations. It does not change much but it's a bit cleaner.

[fra31]

Ok, I ran some tests and the new projections seem equivalent to the old ones, but significantly faster.

Thanks again for the contribution!