Some questions about the paper

[qj1204]

Hello, I am currently reproducing your paper. Regarding Figure 4, I have some questions that I would like to ask you:

1. Is the dataset Imagene-compatible?
2. Besides DiffAttack, which specific surrogate model is attacked by other methods to obtain adversarial samples (which one of Res-50, VGG-19, Mov-v2, Inc-v3, ConvNeXt, and Swin-B)?
3. Are other methods using third-party adversarial attack libraries or running only source code?

There is another question about table 1: The gray background represents a white box attack, but DiffAttack uses the adversarial samples generated by the diffusion model to attack the target model, which should be a black box attack. Why is DiffAttack also set to gray in Table 1?

I would greatly appreciate it if you could reply to me.

[WindVChen]

Hi @qj1204 ,

1. Yes. Visualizations in Fig. 4, Fig. 8, and Fig. 9 are from the ImageNet-Compatible dataset.
2. All are crafted on Inc-v3, including DiffAttack.
3. Please refer to this issue https://github.com/WindVChen/DiffAttack/issues/3#issuecomment-1565459090.
4. There may be some misunderstandings. Although the adversarial samples of DiffAttack are generated by the diffusion model, the optimization of them received guidance from the surrogate classifier (refer to Eq. 3 in the paper). Thus, according to the definition of white-box attacks, when the surrogate model is the same as the target model, the attack is a white-box attack.

Hope these help.

[qj1204]

Thank you for your reply. Now I have a clearer understanding of the DiffAttack's process.

There is one more question. The image is part of the SSA code. Yesterday, I tried to run it, but I don't know if line 108 needs to be deleted when running DI, TI, MI. May I ask which lines you specifically deleted or released at that time?

[WindVChen]

Hi @qj1204 ,

I've written a brief script that you can use to replace the original lines 87-116:

for i in range(num_iter):
    noise = 0

    x_idct = V(x.cuda(), requires_grad=True)

    # DI-FGSM https://arxiv.org/abs/1803.06978
    # output_v3 = model(DI(x_idct))
    output_v3 = model(x_idct)

    loss = F.cross_entropy(output_v3, gt)
    loss.backward()
    noise += x_idct.grad.data

    # TI-FGSM https://arxiv.org/pdf/1904.02884.pdf
    # noise = F.conv2d(noise, T_kernel, bias=None, stride=1, padding=(3, 3), groups=3)

    # MI-FGSM https://arxiv.org/pdf/1710.06081.pdf
    # noise = noise / torch.abs(noise).mean([1, 2, 3], keepdim=True)
    # noise = momentum * grad + noise
    # grad = noise

    x = x + alpha * torch.sign(noise)
    x = clip_by_tensor(x, min, max)

return x.detach()

The script above essentially implements an iterative FGSM (with value clipping). By uncommenting the lines under the respective DI-FGSM, TI-FGSM, and MI-FGSM sections, the code will be modified accordingly. Note that for DI-FGSM, you need to comment out the line output_v3 = model(x_idct).

I hope this helps!

By the way, I noticed that you previously also attached a figure of test results. However, I don't see it now. Has that problem been resolved?

[qj1204]

That picture shows that there is a significant difference between my test results and the results in the paper. Later, I discovered that the order of the images did not match the label, and now I have resolved this issue. However, the current results are slightly higher than those in the paper.

The surrogate model is Inc-v3, and the attack method is S²I-FGSM. My result: Paper's result:

[WindVChen]

That's interesting. From your testing, S²I-FGSM is approximately 4 points less transferable. If there are no fundamental errors in running their source code, this discrepancy might be attributed to differences in the evaluation machines. However, I'm not entirely sure if differences in the machine or environment alone could cause such a significant deterioration in S²I-FGSM performance.

[qj1204]

May I ask how did you set up the model when reproducing S²I-FGSM?

The ignored part is the source code of S²I-FGSM. If I don't modify it to line 150, the resolution cannot be changed to 224. Otherwise, an error will be reported. After adding line 150 and changing the resolution to 224, I obtained the above result.

[WindVChen]

Ah, you may need to include the normalization with L150.

[qj1204]

Sorry, I don't quite understand what you meant. Can you explain it in more detail? Please forgive me for bothering you all this time.

[WindVChen]

You may have a try with this (incorporate the normalization operation into the line 150):

model = torch.nn.Sequential(Normalize(opt.mean, opt.std),
                  models.inception_v3(pretrained=True).eval().cuda())