IndexError in CarliniL2Method

weitianli commented 5 years ago

Describe the bug I implement a simple resnet9 and want to test cw2 result. All is good when I set max_iter=10. But when I set max_iter=1000 that is same value as cw2 original version. The index error comes.

To Reproduce Steps to reproduce the behavior:

Set a simple CarliniL2Method.
cl2m = CarliniL2Method(classifier=ptc, targeted=True, max_iter=1000)
run the cw2 code like in the cw2unittest
See error

Expected behavior No index error and the result comes out.

System information (please complete the following information):

Linux 16.04
Python 3.6
ART version 0.4.0
PyTorch 0.4.1

# First attack
cl2m = CarliniL2Method(classifier=ptc, targeted=True, max_iter=1000)
params={'y': targets}
x_test_adv = cl2m.generate(inputs, **params)
y_pred_adv = np.argmax(ptc.predict(x_test_adv , batch_size=32), axis=1)
logger.info('CW2 Success Rate: %.2f', (sum(adv_label == y_pred_adv) / float(len(adv_label))))
logger.info('Model Acc: %.2f', (sum(adv_original_label == y_pred_adv) / float(len(adv_original_label))))

Traceback (most recent call last):
  File "cw_pytorch.py", line 172, in <module>
    x_test_adv = cl2m.generate(inputs, **params)
  File "/home/weitian/anaconda3/envs/xnor/lib/python3.6/site-packages/art/attacks/carlini.py", line 380, in generate
    x_adv_batch_tanh[active_and_update_adv] = x_adv_batch_tanh[update_adv] + \
IndexError: boolean index did not match indexed array along dimension 0; dimension is 18 but corresponding boolean dimension is 17

mathsinn commented 5 years ago

Thank you @weitianli I'm looking into it.

ririnicolae commented 5 years ago

Hi, @weitianli! Fix is now out, please install from clone for the issue to be solved. Let us know if you encounter any further problems!

mathsinn commented 5 years ago

@weitianli Normally, with the new version of the attack, an max_iter value significantly smaller than 1000 should suffice. Before we were using gradient descent with decayed learning rate for minimizing the CWL2 objective, now we are using binary line search which should converge faster and be more stable. Please let us know if you have any issues with determining the right value for this parameter.

weitianli commented 5 years ago

Thank you very much! I will try it!

weitianli commented 5 years ago

I have a small problem about the max_iter. I found your tutorial code about cw set max_iter is 10. Is it the same effect like Cleverhans tutorial(max_iter=100) or CW(max_iter=1000) original version? Also, I have a small problem about eps in FGSM calculation. Is eps formula in ART is 1*(1/255)? Thank you very much! @mathsinn @ririnicolae

ririnicolae commented 5 years ago

@weitianli What is the problem you encounter with max_iter in C&W? What about the one on the eps value in FGSM? I would say the number of iterations is not strictly equivalent to the other implementations, just that in ART a small number of iterations should suffice. The eps value depends on the data range of your classifier. For example, if your data is normalized between 0 and 1, eps should also be in that range. If your data is between 0 and 255, eps can take values up to 255.

weitianli commented 5 years ago

I got it. Thank you for your kind response.

Trusted-AI / adversarial-robustness-toolbox

IndexError in CarliniL2Method #29