The JSMA implementation in this repository is significantly less effective than as reported in prior work. In Table XIV JSMA appears to succeed 76% of the time. When I run the code in the repository, I get a very similar result: 72.3%.
This should be somewhat surprising given that prior work reports JSMA succeeds above 90% of the time with the same distortion bound of 10% of pixels changed. Unfortunately, Papernot et al. (2016) uses a bound of 14.5% so is not directly comparable, but, in Carlini & Wagner (2016) we re-implemented JSMA and found a 90% attack success rate at 78. Indeed, when I run the JSMA attack from CleverHans on this exact same network (using the approach discussed in #3) I reach 95% attack success rate.
When investigating this just a bit more, I observe that when attacking a solid-black image targeting each possible target label 0 through 9 the code in this repository returns a substantially different adversarial example than the code from CleverHans returns, which this code is based on.
The JSMA implementation in this repository is significantly less effective than as reported in prior work. In Table XIV JSMA appears to succeed 76% of the time. When I run the code in the repository, I get a very similar result: 72.3%.
This should be somewhat surprising given that prior work reports JSMA succeeds above 90% of the time with the same distortion bound of 10% of pixels changed. Unfortunately, Papernot et al. (2016) uses a bound of 14.5% so is not directly comparable, but, in Carlini & Wagner (2016) we re-implemented JSMA and found a 90% attack success rate at 78. Indeed, when I run the JSMA attack from CleverHans on this exact same network (using the approach discussed in #3) I reach 95% attack success rate.
When investigating this just a bit more, I observe that when attacking a solid-black image targeting each possible target label 0 through 9 the code in this repository returns a substantially different adversarial example than the code from CleverHans returns, which this code is based on.