Dropout needs to be turned off explicitely when generating attacks

[spalaciob]

Following the README, trying to execute: python craft_adv_samples.py -d=mnist -a=fgsm got me the following error:

Traceback (most recent call last):
   File "craft_adv_samples.py", line 124, in <module>
    main(args)
  File "craft_adv_samples.py", line 99, in main
    args.batch_size)
  File "craft_adv_samples.py", line 40, in craft_one_type
    clip_max=1., batch_size=batch_size
  File "/home/user/detecting-adversarial-samples/src/attacks.py", line 185, in fast_gradient_sign_method
    [X, Y], args={'batch_size': batch_size}
  File "/home/user/detecting-adversarial-samples/scripts/src/cleverhans/cleverhans/utils_tf.py", line 350, in batch_eval
(...)
tensorflow.python.framework.errors_impl.InvalidArgumentError: You must feed a value for placeholder tensor 'dropout_1/keras_learning_phase' with dtype bool
         [[Node: dropout_1/keras_learning_phase = Placeholder[dtype=DT_BOOL, shape=[], _device="/job:localhost/replica:0/task:0/gpu:0"]()]]

It seems that the cleverhans function batch_eval needs the dropout layers to be explicitely turned off. For the specific case of the tutorial, modifying src/attacks.py in line 185 as follows:

    X_adv, = batch_eval(
        sess, [x, y], [adv_x],
-       [X, Y], args={'batch_size': batch_size}
+       [X, Y], args={'batch_size': batch_size}, feed={'dropout_1/keras_learning_phase:0': False}
        )

solves the problem.

I suppose that other models won't necessarily have that exact name for their dropout layers (haven't tested them) so I can't really vouch for this solution as an always working patch.

[rfeinman]

@spalaciob thanks for catching this. What version of cleverhans are you using? My guess is that you are using cleverhans 2, which was released a few days ago. I have not had time to step through this repository and update all of the code to go along with cleverhans 2. For the time being, I will likely update the requirements.txt file to require cleverhans == 1.0.0. If you'd like to step through and help out, I'd be happy to look through your changes and merge.

[spalaciob]

Found a better way to solve the problem :) Keras has a global variable to tell whether it's on training or testing mode (K.training_phase in case someone's wondering). Setting it to 0 (testing mode) solves the issue. Pull request has been sent already.

[rfeinman]

Thanks for the PR. Closing this issue for now.