TF1: How do labels get modified by batch_datapoint_idcs?

[giorgiopiras]

Introduction

Hi @fra31 , thanks for your work on this repo. I was trying to run autoattack on Tensorflow 1x, but i'm facing the following error:

using standard version including apgd-ce, apgd-t, fab-t, square
Warning: the check for dynamic defenses is not currently supported
initial accuracy: 10.00%
Traceback (most recent call last):
  File "/home/gpiras/others/data/ANP_VS/Attack_pipeline.py", line 74, in <module>
    x_adv = adversary.run_standard_evaluation(torch_tx[:n_ex], torch_ty[:n_ex])
  File "/home/gpiras/miniconda3/envs/anpvs_4/lib/python3.6/site-packages/autoattack/autoattack.py", line 156, in run_standard_evaluation
    adv_curr = self.apgd.perturb(x, y) #cheap=True
  File "/home/gpiras/miniconda3/envs/anpvs_4/lib/python3.6/site-packages/autoattack/autopgd_base.py", line 473, in perturb
    acc = y_pred == y
  File "/home/gpiras/miniconda3/envs/anpvs_4/lib/python3.6/site-packages/torch/tensor.py", line 27, in wrapped
    return f(*args, **kwargs)
RuntimeError: The size of tensor a (20) must match the size of tensor b (2) at non-singleton dimension 0

Where does the error come from?

I was debugging the code, and apparently the problem(?) comes from line 141 in autoattack/autoattack.py. y = y_orig[batch_datapoint_idcs].clone().to(self.device) In this line, y becomes a vector with only the elements from y_orig indicized by batch_datapoint_idcs (normally two or four elements). Therefore, the size mismatch between tensors. I was wondering whether this is normal.

Here's the code from which I am launching autoattack:

import ...

#attack settings 
norm = 'Linf' 
version = 'standard'
epsilon = .03
batch_size = 20
n_ex = 20
n_test_batches = 1

#data 
...
x_test, y_test = input_fn(False, batch_size)

#create model
...

# import model 
with tf.Session() as sess:
    # create the graph structure from the meta file
    saver=tf.train.import_meta_graph("...model.meta")
    # restore the session and the parameters
    saver.restore(sess, "...model")
    sess.run(tf.global_variables_initializer())
    graph = tf.get_default_graph()

    # given the batches, tak the test data 
    for it in range(1, n_test_batches+1): 
        tx, ty = sess.run([x_test, y_test])

    # take the logits for ModelAdapter
    x, temp_x = net.__call__(tx, train=False)  
    # before running the sess, print(x) to see the logits opname
    print(x)
    val = sess.run('add_805:0') 
    logits = tf.convert_to_tensor(val)

    # adapt labels 
    resized_y = sess.run([tf.where(condition=vect) for vect in ty])
    # flatten the labels
    flattened_y = []
    for el in resized_y: 
      flattened_y.extend(el[0])

    # prepare data to be used by run_standard_evaluation
    torch_tx = torch.from_numpy(tx) 
    torch_ty = torch.Tensor(flattened_y).type(torch.int64) 

    # prepare placeholders for ModelAdapter
    y_input = tf.placeholder(tf.int64, shape=[None])
    x_input = tf.placeholder(dtype=tf.float64, shape=[None,None,None,None])  #tx #unashable type nparray

    # setting the autoattack
    model_adapted = utils_tf.ModelAdapter(logits, x_input, y_input, sess)
    adversary = AutoAttack(model_adapted, norm='Linf', eps=epsilon, version=version, is_tf_model=True)
    x_adv = adversary.run_standard_evaluation(torch_tx[:n_ex], torch_ty[:n_ex], bs=batch_size)

Can you help me understand if this is a bug or if I am missing something please? Thank you for your help

[fra31]

Hi,

it's a bit hard to say without more context, but I noticed that clean accuracy is only 10%: is this expected? Also, the logits passed to ModelAdapter should be the tensor which returns the logits depending on the placeholder of the input, but from the snippet above it seems that x_input and logits are not related: is it possible that logits is just a fixed vector instead the output of the network?

[giorgiopiras]

Hi, sorry for the late reply. You were right by the way, I was feeding the ModelAdapter with fixed logits, I should have now solved by using something like:

    y_input = tf.placeholder(tf.int64, shape=[None])
    x_input = tf.placeholder(dtype=tf.float32, shape=x_test.shape) 

    # take the logits given the call function
    logits, _ = net.__call__(x_input, train=False)   

    # setting the autoattack
    model_adapted = utils_tf.ModelAdapter(logits, x_input, y_input, sess)
    adversary = AutoAttack(model_adapted, norm='Linf', eps=epsilon, version=version, is_tf_model=True)
    x_adv = adversary.run_standard_evaluation(torch_tx, torch_ty, bs=batch_size)

This gives me new errors though. To give you a bit of context, I am using CIFAR10 input samples and I have tested this model before without using autoattack. Everything runs in Tensorflow 1.12.0
Anyway, what happens is this:

  File "/home/gpiras/others/data/ANP_VS/Attack_pipeline.py", line 72, in <module>
    x_adv = adversary.run_standard_evaluation(torch_tx, torch_ty, bs=batch_size)
  File "/home/gpiras/miniconda3/envs/anpvs_4/lib/python3.6/site-packages/autoattack/autoattack.py", line 89, in run_standard_evaluation
    y_orig[:bs].to(self.device), bs=bs, logger=self.logger)
  File "/home/gpiras/miniconda3/envs/anpvs_4/lib/python3.6/site-packages/autoattack/checks.py", line 24, in check_randomized
    output = model(x)
  File "/home/gpiras/miniconda3/envs/anpvs_4/lib/python3.6/site-packages/autoattack/autoattack.py", line 75, in get_logits
    return self.model.predict(x)
  File "/home/gpiras/miniconda3/envs/anpvs_4/lib/python3.6/site-packages/autoattack/utils_tf.py", line 45, in predict
    y = self.sess.run(self.logits, {self.x_input: x2})
  File "/home/gpiras/miniconda3/envs/anpvs_4/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 929, in run
    run_metadata_ptr)
  File "/home/gpiras/miniconda3/envs/anpvs_4/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1128, in _run
    str(subfeed_t.get_shape())))
ValueError: Cannot feed value of shape (20, 32, 32, 3) for Tensor 'Placeholder_4:0', which has shape '(?, 3, 32, 32)'

Basically the problem comes from the placeholder I create, which has shape (?, 3, 32, 32) and the input that autoattack feeds the placeholder with, which has shape (20,32,32,3). This input fed by autoattack though, should be torch_tx, which yet has shape (20,3,32,32), the problem is autoattack in line 45 of autoattack/utils_tf:

x2 = np.moveaxis(x.cpu().numpy(), 1, 3)

y = self.sess.run(self.logits, {self.x_input: x2})

By moving the axis the input gets modified and does not match the placeholder shape. Do you have any idea how I can manage this? I tried modifying the x2 creation by removing the moveaxis function, but it gives me simply 0% initial accuracy and perturbations won't work. Thanks

[fra31]

The attacks expect images with channel first format (i.e. (batch size, 3, 32, 32) for CIFAR-10), since that's the standard one returned by PyTorch data loader. Since most of TensorFlow models take input in channel last format instead, the interface for TF1.X expects that and moves the axes accordingly. Which data format is used by your classifier? If it's channel last, creating a placeholder with shape [None, 32, 32, 3] should work. Otherwise, I guess you could either still create a placeholder with channel last format and add a reshaping of the input to your model definition, or alternatively modify the functions in utils_tf.py. Also, here you can find an example with TFv1.

[giorgiopiras]

The input data my model takes is channel first, which is the shape I am giving to the placeholder accordingly. By changing the functions inside of utils_tf.py as x2 = np.array(x.cpu()) it still reiterates back to the original error of this issue from line 141 in autoattack/autoattack.py

setting parameters for standard version
using standard version including apgd-ce, apgd-t, fab-t, square
Warning: the check for dynamic defenses is not currently supported
initial accuracy: 20.00%
This is the y printed after line 144 tensor([6, 6, 6, 6], device='cuda:0')
/home/gpiras/miniconda3/envs/anpvs_4/lib/python3.6/site-packages/autoattack/autopgd_base.py:502: UserWarning: This overload of nonzero is deprecated:
        nonzero()
Consider using one of the following signatures instead:
        nonzero(*, bool as_tuple) (Triggered internally at  /pytorch/torch/csrc/utils/python_arg_parser.cpp:882.)
  ind_to_fool = acc.nonzero().squeeze()
Traceback (most recent call last):
  File "/home/gpiras/others/data/ANP_VS/Attack_pipeline.py", line 74, in <module>
    x_adv = adversary.run_standard_evaluation(torch_tx, torch_ty, bs=batch_size)
  File "/home/gpiras/miniconda3/envs/anpvs_4/lib/python3.6/site-packages/autoattack/autoattack.py", line 155, in run_standard_evaluation
    adv_curr = self.apgd.perturb(x, y) #cheap=True
  File "/home/gpiras/miniconda3/envs/anpvs_4/lib/python3.6/site-packages/autoattack/autopgd_base.py", line 511, in perturb
    res_curr = self.attack_single_run(x_to_fool, y_to_fool)
  File "/home/gpiras/miniconda3/envs/anpvs_4/lib/python3.6/site-packages/autoattack/autopgd_base.py", line 280, in attack_single_run
    grad += grad_curr
RuntimeError: The size of tensor a (32) must match the size of tensor b (3) at non-singleton dimension 2

Do you know by any chance what might be happening here? I honestly can't figure

[fra31]

That's because if the data is already in channel first format the gradients do not need to be reshaped e.g. in https://github.com/fra31/auto-attack/blob/b7f560b229145e6e90613cd3ce98cad6a94bd623/autoattack/utils_tf.py#L52 I think adding the reshaping directly in the model definition so that the placeholder is in channel last format could be an easier solution.

[giorgiopiras]

Ok I see, but that would mean retraining the model from scratch I guess. Also, this model was written in TF1.x by others and they did not use Keras support at all unfortunately, so this should make the process even harder I believe. Counter-intuitively I think it would be easier to reshape gradients in utils_tf.py

[fra31]

I guess one can just redefine the logits with something like

def forward(x):
  z = tf.reshape(x, [-1, 3, 32, 32]) # or tf.transpose()?
  y, _ = net.__call__(z, train=False)
  return y

logits = forward(x_input)

with x_test in channel last format (this is not tested though). Otherwise yes, you can modify the other functions.

[giorgiopiras]

Thanks, I managed by modifying the functions.