Invalid configuration for square attack

[CNOCycle]

Hi authors,

I encountered an unexpected error when I was running L-2 version of Square attack. After my detail investigation, I would guess that the problem may be caused by an improper configuration s.

The parameter s is configured at line line 250, 335 and 435 for L-inf, L-2 and L-1 version respectively.

# line 250 for L-inf
s = max(int(round(math.sqrt(p * n_features / c))), 1)
# line 335 for L-2
s = max(int(round(math.sqrt(p * n_features / c))), 3)
if s % 2 == 0:
    s += 1
# line 435 for L-1
s = max(int(round(math.sqrt(p * n_features / c))), 3)
if s % 2 == 0:
    s += 1

Then, square attack randomly selectsvh and vw immediately.

vh = self.random_int(0, h - s)
vw = self.random_int(0, w - s)

The function random_init is defined around line 117:

def random_int(self, low=0, high=1, shape=[1]):
    t = low + (high - low) * torch.rand(shape).to(self.device)
    return t.long()

The parameter high should be larger than low literally. However, this case may not always be true.

If $s>h$, then low > high. The following is the detailed explanation: Without loss of generality, we assume that w > h, , where w and h are the given image's input size. Since p=0.8 by default and n_features = c * h * w, we know that $s=int(round((\sqrt{0.8wh}))$.

The equation $s>h$ can be simplified as follows: $s>h$ -> $\sqrt{0.8wh}>h$ -> $0.8w>h$

Or we can assume that w=500. We immediately have a constraint $int(round((\sqrt{0.8*500h})) > h$ -> $20\sqrt{h} > h$ -> $400 > h$.

This result suggests that square attack only works for images have specific aspect ratio. For CIFAR10 and CIFAR100, the above constraint does not hold, but images on ImageNet have vary aspect ratio.

The following is the minimal test case:

# git clone https://github.com/fra31/auto-attack.git
# git log -1 | head -n 1
# > commit e8e64bd17a0ad351d5c7390d3ce1a0840ae50a96

#%%
import logging
import os
from argparse import ArgumentParser

import torch
from torchvision import transforms, models

from autoattack import AutoAttack

#%%
def arg_parser(parser):

    parser.add_argument('--norm', type=str, default='L2')
    parser.add_argument('--epsilon', type=float, default=4./255.)
    parser.add_argument('--batch_size', type=int, default=1)
    parser.add_argument('--img_w', type=int, default=400)
    parser.add_argument('--img_h', type=int, default=500)

    args, unknown = parser.parse_known_args()

    return args, unknown

#%%
if __name__ == '__main__':

    # get arguments
    parser = ArgumentParser()
    args, unknown = arg_parser(parser)

    # set logging level
    logging.basicConfig(level=logging.INFO,
                        handlers=[logging.StreamHandler()],
                        datefmt='%Y-%m-%d %H:%M:%S',
                        format='%(asctime)s.%(msecs)03d [%(levelname)s] %(module)s:%(funcName)s:%(lineno)d | %(message)s')

    # set and print hyper-parameters
    gpu_vars     = os.environ.get("CUDA_VISIBLE_DEVICES", "")
    logging.info('[ GPU ] CUDA_VISIBLE_DEVICES: {:s}'.format(gpu_vars))
    logging.info('[torch] Pytorch version: {:s}'.format(torch.__version__) )
    logging.info('[param] Listing hyper-parameters...')
    logging.info('[param] dataset    : {:s}'.format('imagenet') )
    logging.info('[param] norm       : {:s}'.format(args.norm) )
    logging.info('[param] epsilon    : {:f}'.format(args.epsilon) )
    logging.info('[param] image_size : ({:d}, {:d})'.format(args.img_w, args.img_h) )

    # enable auto-tune
    torch.backends.cudnn.benchmark = True

    # load pytorch's pre-trained Resnet-50
    model = models.resnet50(pretrained=True)

    # move model to GPU and convert to eval mode
    model.cuda()
    model.eval()
    model = torch.jit.script(model)
    #model = torch.jit.trace(model, (x_test[0:args.batch_size, :]))

    # load image
    #from PIL import Image
    #img_path = '/shared/dataset-imagenet/val/n07892512/ILSVRC2012_val_00031066.JPEG'
    #img = Image.open(img_path)
    #(w, h) = img.size
    #logging.info('orginal size = ({:d}, {:d})'.format(w, h))
    #img = img.resize((args.img_w, args.img_h))
    #(w, h) = img.size
    #logging.info('resized size = ({:d}, {:d})'.format(w, h))
    #img_tensor = transforms.functional.pil_to_tensor(img)
    #img_tensor = img_tensor.to(torch.float32)
    #img_tensor = img_tensor / 255.0

    # load image - Create a blank image from random noise
    img_tensor = torch.rand((3, args.img_w, args.img_h), dtype=torch.float32)

    x_test = torch.unsqueeze(img_tensor, 0)
    x_test = x_test.to("cuda")

    y_test = torch.argmax(model(x_test))
    y_test = torch.unsqueeze(y_test, 0)
    y_test = y_test.to("cuda")

    # load attack    
    adversary = AutoAttack(model, norm=args.norm, eps=args.epsilon, log_path=None)
    adversary.attacks_to_run = ['square']

    # run attack and save images
    with torch.no_grad():
        adv_complete = adversary.run_standard_evaluation(x_test, y_test, bs=args.batch_size)

The following is the output for a normal case (command python test_square_attack.py --img_w 401):

2022-09-10 08:14:49.298 [INFO] test_square_attack:<module>:42 | [ GPU ] CUDA_VISIBLE_DEVICES: 
2022-09-10 08:14:49.298 [INFO] test_square_attack:<module>:43 | [torch] Pytorch version: 1.12.0+cu113
2022-09-10 08:14:49.298 [INFO] test_square_attack:<module>:44 | [param] Listing hyper-parameters...
2022-09-10 08:14:49.298 [INFO] test_square_attack:<module>:45 | [param] dataset    : imagenet
2022-09-10 08:14:49.298 [INFO] test_square_attack:<module>:46 | [param] norm       : L2
2022-09-10 08:14:49.298 [INFO] test_square_attack:<module>:47 | [param] epsilon    : 0.015686
2022-09-10 08:14:49.298 [INFO] test_square_attack:<module>:48 | [param] image_size : (401, 500)
setting parameters for standard version
using standard version including square
initial accuracy: 100.00%
square - 1/1 - 0 out of 1 successfully perturbed
robust accuracy after SQUARE: 100.00% (total time 96.2 s)
max L2 perturbation: 0.00000, nan in tensor: 0, max: 1.00000, min: 0.00000
robust accuracy: 100.00%

The following is the output for an invalid case (command python test_square_attack.py --img_w 375):

2022-09-10 08:16:48.961 [INFO] test_square_attack:<module>:42 | [ GPU ] CUDA_VISIBLE_DEVICES: 
2022-09-10 08:16:48.961 [INFO] test_square_attack:<module>:43 | [torch] Pytorch version: 1.12.0+cu113
2022-09-10 08:16:48.961 [INFO] test_square_attack:<module>:44 | [param] Listing hyper-parameters...
2022-09-10 08:16:48.961 [INFO] test_square_attack:<module>:45 | [param] dataset    : imagenet
2022-09-10 08:16:48.961 [INFO] test_square_attack:<module>:46 | [param] norm       : L2
2022-09-10 08:16:48.961 [INFO] test_square_attack:<module>:47 | [param] epsilon    : 0.015686
2022-09-10 08:16:48.961 [INFO] test_square_attack:<module>:48 | [param] image_size : (375, 500)
setting parameters for standard version
using standard version including square
initial accuracy: 100.00%
Traceback (most recent call last):
  File "test_square_attack.py", line 91, in <module>
    adv_complete = adversary.run_standard_evaluation(x_test, y_test, bs=args.batch_size)
  File "/tmp/auto-attack/autoattack/autoattack.py", line 169, in run_standard_evaluation
    adv_curr = self.square.perturb(x, y)
  File "/tmp/auto-attack/autoattack/square.py", line 587, in perturb
    _, adv_curr = self.attack_single_run(x_to_fool, y_to_fool)
  File "/tmp/auto-attack/autoattack/square.py", line 362, in attack_single_run
    new_deltas += old_deltas
RuntimeError: The size of tensor a (387) must match the size of tensor b (375) at non-singleton dimension 2

The following is the output for a corner case (command python test_square_attack.py --img_w 400):

2022-09-10 08:18:18.332 [INFO] test_square_attack:<module>:42 | [ GPU ] CUDA_VISIBLE_DEVICES: 
2022-09-10 08:18:18.332 [INFO] test_square_attack:<module>:43 | [torch] Pytorch version: 1.12.0+cu113
2022-09-10 08:18:18.332 [INFO] test_square_attack:<module>:44 | [param] Listing hyper-parameters...
2022-09-10 08:18:18.332 [INFO] test_square_attack:<module>:45 | [param] dataset    : imagenet
2022-09-10 08:18:18.332 [INFO] test_square_attack:<module>:46 | [param] norm       : L2
2022-09-10 08:18:18.332 [INFO] test_square_attack:<module>:47 | [param] epsilon    : 0.015686
2022-09-10 08:18:18.332 [INFO] test_square_attack:<module>:48 | [param] image_size : (400, 500)
setting parameters for standard version
using standard version including square
initial accuracy: 100.00%
Traceback (most recent call last):
  File "test_square_attack.py", line 91, in <module>
    adv_complete = adversary.run_standard_evaluation(x_test, y_test, bs=args.batch_size)
  File "/tmp/auto-attack/autoattack/autoattack.py", line 169, in run_standard_evaluation
    adv_curr = self.square.perturb(x, y)
  File "/tmp/auto-attack/autoattack/square.py", line 587, in perturb
    _, adv_curr = self.attack_single_run(x_to_fool, y_to_fool)
  File "/tmp/auto-attack/autoattack/square.py", line 362, in attack_single_run
    new_deltas += old_deltas
RuntimeError: The size of tensor a (401) must match the size of tensor b (400) at non-singleton dimension 2

[fra31]

Hi,

thanks for the detailed analysis, I think I never used it for non-squared images. I'd say that the minimal change to handle such case could be clipping the value of s, e.g. s = min(s, min(h, w)). This would preserve the square-shaped updates and be compatible with the rest of the algorithm.

The drawback could be that if e.g. $h \ll w$ only updates with small area are allowed, meaning a slower progress of the optimization at the beginning of the algorithm. Then, another option could be to roughly preserve the number of pixels $s^2$ with a rectangular update (for example mimicking the aspect ratio of the image), but this would require a bigger change in the code.

Would the first solution work for you?

[CNOCycle]

Hi,

Thanks for your quick reply. Since function p_selection will decrease p gradually, the constraint I mentioned generally appears at the beginning. Currently, my workaround is as same as your first proposed solution. However, if huge modifications are not allowable, I will suggest that square attack should give a aspect ratio warning to users when detecting non-square images.