search

cleverhans-lab / cleverhans

An adversarial example library for constructing attacks, building defenses, and benchmarking both
MIT License
6.2k stars 1.39k forks source link

Flaky test #1145

Open loopylangur opened 4 years ago

loopylangur commented 4 years ago

The TestSaliencyMapMethod.test_generate_np_targeted_gives_adversarial_example in tests_tf\test_attacks.py fails intermittently with the following error:

>     self.assertTrue(worked > .9)
E     AssertionError: False is not true

The value of worked in this case is 0.9

Full error log:

============================================================================================== test session starts ==============================================================================================
platform linux -- Python 3.7.5, pytest-5.3.2, py-1.8.1, pluggy-0.13.1
rootdir: cleverhans
collected 1 item

tests_tf/test_attacks.py F                                                                                                                                                                                [100%]

=================================================================================================== FAILURES ====================================================================================================
___________________________________________________________________ TestSaliencyMapMethod.test_generate_np_targeted_gives_adversarial_example ___________________________________________________________________

self = <test_attacks.TestSaliencyMapMethod testMethod=test_generate_np_targeted_gives_adversarial_example>

    def test_generate_np_targeted_gives_adversarial_example(self):
      x_val = np.random.rand(10, 1000)
      x_val = np.array(x_val, dtype=np.float32)

      feed_labs = np.zeros((10, 10))
      feed_labs[np.arange(10), np.random.randint(0, 9, 10)] = 1
      x_adv = self.attack.generate_np(x_val,
                                      clip_min=-5., clip_max=5.,
                                      y_target=feed_labs)
      new_labs = np.argmax(self.sess.run(self.model.get_logits(x_adv)), axis=1)

      worked = np.mean(np.argmax(feed_labs, axis=1) == new_labs)
>     self.assertTrue(worked > .9)
E     AssertionError: False is not true

tests_tf/test_attacks.py:1243: AssertionError
--------------------------------------------------------------------------------------------- Captured stdout call ----------------------------------------------------------------------------------------------
test_attacks.TestSaliencyMapMethod.test_generate_np_targeted_gives_adversarial_example took 11.888648509979248 seconds
--------------------------------------------------------------------------------------------- Captured stderr call ----------------------------------------------------------------------------------------------
WARNING:tensorflow:From cleverhans/tests_tf/test_attacks.py:1216: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.

WARNING:tensorflow:From cleverhans/tests_tf/test_attacks.py:1219: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

2020-01-18 11:51:38.216156: I tensorflow/core/platform/cpu_feature_guard.cc:145] This TensorFlow binary is optimized with Intel(R) MKL-DNN to use the following CPU instructions in performance critical operations:  SSE4.1 SSE4.2 AVX AVX2 FMA
To enable them in non-MKL-DNN operations, rebuild TensorFlow with the appropriate compiler flags.
2020-01-18 11:51:38.223348: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2993350000 Hz
2020-01-18 11:51:38.224200: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x564d26609fe0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-01-18 11:51:38.224218: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
OMP: Info #212: KMP_AFFINITY: decoding x2APIC ids.
OMP: Info #210: KMP_AFFINITY: Affinity capable, using global cpuid leaf 11 info
OMP: Info #154: KMP_AFFINITY: Initial OS proc set respected: 0-23
OMP: Info #156: KMP_AFFINITY: 24 available OS procs
OMP: Info #157: KMP_AFFINITY: Uniform topology
OMP: Info #179: KMP_AFFINITY: 1 packages x 12 cores/pkg x 2 threads/core (12 total cores)
OMP: Info #214: KMP_AFFINITY: OS proc to physical thread map:
OMP: Info #171: KMP_AFFINITY: OS proc 0 maps to package 0 core 0 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 12 maps to package 0 core 0 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 1 maps to package 0 core 1 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 13 maps to package 0 core 1 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 2 maps to package 0 core 2 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 14 maps to package 0 core 2 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 3 maps to package 0 core 3 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 15 maps to package 0 core 3 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 4 maps to package 0 core 4 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 16 maps to package 0 core 4 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 5 maps to package 0 core 5 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 17 maps to package 0 core 5 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 6 maps to package 0 core 8 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 18 maps to package 0 core 8 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 7 maps to package 0 core 9 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 19 maps to package 0 core 9 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 8 maps to package 0 core 10 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 20 maps to package 0 core 10 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 9 maps to package 0 core 11 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 21 maps to package 0 core 11 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 10 maps to package 0 core 12 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 22 maps to package 0 core 12 thread 1 
OMP: Info #171: KMP_AFFINITY: OS proc 11 maps to package 0 core 13 thread 0 
OMP: Info #171: KMP_AFFINITY: OS proc 23 maps to package 0 core 13 thread 1 
OMP: Info #250: KMP_AFFINITY: pid 2177 tid 2177 thread 0 bound to OS proc set 0
2020-01-18 11:51:38.227585: I tensorflow/core/common_runtime/process_util.cc:115] Creating new thread pool with default inter op setting: 2. Tune using inter_op_parallelism_threads for best performance.
WARNING:tensorflow:From cleverhans/tests_tf/test_attacks.py:1226: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

WARNING:tensorflow:From cleverhans/tests_tf/test_attacks.py:89: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

WARNING:tensorflow:From cleverhans/tests_tf/test_attacks.py:89: The name tf.AUTO_REUSE is deprecated. Please use tf.compat.v1.AUTO_REUSE instead.

WARNING:tensorflow:From python3.7/site-packages/tensorflow_core/contrib/layers/python/layers/layers.py:1634: flatten (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.flatten instead.
WARNING:tensorflow:From python3.7/site-packages/tensorflow_core/python/layers/core.py:332: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `layer.__call__` method instead.
WARNING:tensorflow:From cleverhans/tests_tf/test_attacks.py:1227: The name tf.global_variables_initializer is deprecated. Please use tf.compat.v1.global_variables_initializer instead.

[INFO 2020-01-18 11:51:38,282 cleverhans] Constructing new graph for attack SaliencyMapMethod
WARNING:tensorflow:From cleverhans/cleverhans/attacks/saliency_map_method.py:246: The name tf.mod is deprecated. Please use tf.math.mod instead.

WARNING:tensorflow:From cleverhans/cleverhans/attacks/saliency_map_method.py:247: The name tf.floordiv is deprecated. Please use tf.math.floordiv instead.

OMP: Info #250: KMP_AFFINITY: pid 2177 tid 2255 thread 1 bound to OS proc set 1
OMP: Info #250: KMP_AFFINITY: pid 2177 tid 2257 thread 2 bound to OS proc set 2
OMP: Info #250: KMP_AFFINITY: pid 2177 tid 2259 thread 4 bound to OS proc set 4
OMP: Info #250: KMP_AFFINITY: pid 2177 tid 2258 thread 3 bound to OS proc set 3
OMP: Info #250: KMP_AFFINITY: pid 2177 tid 2260 thread 5 bound to OS proc set 5
OMP: Info #250: KMP_AFFINITY: pid 2177 tid 2261 thread 6 bound to OS proc set 6
OMP: Info #250: KMP_AFFINITY: pid 2177 tid 2262 thread 7 bound to OS proc set 7
OMP: Info #250: KMP_AFFINITY: pid 2177 tid 2263 thread 8 bound to OS proc set 8
OMP: Info #250: KMP_AFFINITY: pid 2177 tid 2264 thread 9 bound to OS proc set 9
OMP: Info #250: KMP_AFFINITY: pid 2177 tid 2265 thread 10 bound to OS proc set 10
OMP: Info #250: KMP_AFFINITY: pid 2177 tid 2266 thread 11 bound to OS proc set 11
OMP: Info #250: KMP_AFFINITY: pid 2177 tid 2268 thread 13 bound to OS proc set 13
OMP: Info #250: KMP_AFFINITY: pid 2177 tid 2267 thread 12 bound to OS proc set 12
OMP: Info #250: KMP_AFFINITY: pid 2177 tid 2269 thread 14 bound to OS proc set 14
OMP: Info #250: KMP_AFFINITY: pid 2177 tid 2270 thread 15 bound to OS proc set 15
OMP: Info #250: KMP_AFFINITY: pid 2177 tid 2271 thread 16 bound to OS proc set 16
OMP: Info #250: KMP_AFFINITY: pid 2177 tid 2272 thread 17 bound to OS proc set 17
OMP: Info #250: KMP_AFFINITY: pid 2177 tid 2273 thread 18 bound to OS proc set 18
OMP: Info #250: KMP_AFFINITY: pid 2177 tid 2274 thread 19 bound to OS proc set 19
OMP: Info #250: KMP_AFFINITY: pid 2177 tid 2275 thread 20 bound to OS proc set 20
OMP: Info #250: KMP_AFFINITY: pid 2177 tid 2276 thread 21 bound to OS proc set 21
OMP: Info #250: KMP_AFFINITY: pid 2177 tid 2277 thread 22 bound to OS proc set 22
OMP: Info #250: KMP_AFFINITY: pid 2177 tid 2278 thread 23 bound to OS proc set 23
OMP: Info #250: KMP_AFFINITY: pid 2177 tid 2279 thread 24 bound to OS proc set 0
OMP: Info #250: KMP_AFFINITY: pid 2177 tid 2254 thread 25 bound to OS proc set 1
OMP: Info #250: KMP_AFFINITY: pid 2177 tid 2280 thread 26 bound to OS proc set 2
OMP: Info #250: KMP_AFFINITY: pid 2177 tid 2281 thread 27 bound to OS proc set 3
OMP: Info #250: KMP_AFFINITY: pid 2177 tid 2282 thread 28 bound to OS proc set 4
OMP: Info #250: KMP_AFFINITY: pid 2177 tid 2283 thread 29 bound to OS proc set 5
OMP: Info #250: KMP_AFFINITY: pid 2177 tid 2284 thread 30 bound to OS proc set 6
OMP: Info #250: KMP_AFFINITY: pid 2177 tid 2285 thread 31 bound to OS proc set 7
OMP: Info #250: KMP_AFFINITY: pid 2177 tid 2286 thread 32 bound to OS proc set 8
OMP: Info #250: KMP_AFFINITY: pid 2177 tid 2287 thread 33 bound to OS proc set 9
OMP: Info #250: KMP_AFFINITY: pid 2177 tid 2288 thread 34 bound to OS proc set 10
OMP: Info #250: KMP_AFFINITY: pid 2177 tid 2289 thread 35 bound to OS proc set 11
OMP: Info #250: KMP_AFFINITY: pid 2177 tid 2291 thread 37 bound to OS proc set 13
OMP: Info #250: KMP_AFFINITY: pid 2177 tid 2290 thread 36 bound to OS proc set 12
OMP: Info #250: KMP_AFFINITY: pid 2177 tid 2292 thread 38 bound to OS proc set 14
OMP: Info #250: KMP_AFFINITY: pid 2177 tid 2293 thread 39 bound to OS proc set 15
OMP: Info #250: KMP_AFFINITY: pid 2177 tid 2294 thread 40 bound to OS proc set 16
OMP: Info #250: KMP_AFFINITY: pid 2177 tid 2295 thread 41 bound to OS proc set 17
OMP: Info #250: KMP_AFFINITY: pid 2177 tid 2297 thread 43 bound to OS proc set 19
OMP: Info #250: KMP_AFFINITY: pid 2177 tid 2296 thread 42 bound to OS proc set 18
OMP: Info #250: KMP_AFFINITY: pid 2177 tid 2298 thread 44 bound to OS proc set 20
OMP: Info #250: KMP_AFFINITY: pid 2177 tid 2299 thread 45 bound to OS proc set 21
OMP: Info #250: KMP_AFFINITY: pid 2177 tid 2300 thread 46 bound to OS proc set 22
OMP: Info #250: KMP_AFFINITY: pid 2177 tid 2301 thread 47 bound to OS proc set 23
OMP: Info #250: KMP_AFFINITY: pid 2177 tid 2302 thread 48 bound to OS proc set 0
----------------------------------------------------------------------------------------------- Captured log call -----------------------------------------------------------------------------------------------
WARNING  tensorflow:module_wrapper.py:139 From cleverhans/tests_tf/test_attacks.py:1216: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.

WARNING  tensorflow:module_wrapper.py:139 From cleverhans/tests_tf/test_attacks.py:1219: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.

WARNING  tensorflow:module_wrapper.py:139 From cleverhans/tests_tf/test_attacks.py:1226: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.

WARNING  tensorflow:module_wrapper.py:139 From cleverhans/tests_tf/test_attacks.py:89: The name tf.variable_scope is deprecated. Please use tf.compat.v1.variable_scope instead.

WARNING  tensorflow:module_wrapper.py:139 From cleverhans/tests_tf/test_attacks.py:89: The name tf.AUTO_REUSE is deprecated. Please use tf.compat.v1.AUTO_REUSE instead.

WARNING  tensorflow:deprecation.py:323 From python3.7/site-packages/tensorflow_core/contrib/layers/python/layers/layers.py:1634: flatten (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.
Instructions for updating:
Use keras.layers.flatten instead.
WARNING  tensorflow:deprecation.py:323 From python3.7/site-packages/tensorflow_core/python/layers/core.py:332: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.
Instructions for updating:
Please use `layer.__call__` method instead.
WARNING  tensorflow:module_wrapper.py:139 From cleverhans/tests_tf/test_attacks.py:1227: The name tf.global_variables_initializer is deprecated. Please use tf.compat.v1.global_variables_initializer instead.

INFO     cleverhans:attack.py:127 Constructing new graph for attack SaliencyMapMethod
WARNING  tensorflow:module_wrapper.py:139 From cleverhans/cleverhans/attacks/saliency_map_method.py:246: The name tf.mod is deprecated. Please use tf.math.mod instead.

WARNING  tensorflow:module_wrapper.py:139 From cleverhans/cleverhans/attacks/saliency_map_method.py:247: The name tf.floordiv is deprecated. Please use tf.math.floordiv instead.
=============================================================================================== warnings summary ================================================================================================
python3.7/site-packages/nose/importer.py:12
  python3.7/site-packages/nose/importer.py:12: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
    from imp import find_module, load_module, acquire_lock, release_lock

python3.7/site-packages/tensorflow_core/contrib/learn/python/learn/learn_io/generator_io.py:26
  python3.7/site-packages/tensorflow_core/contrib/learn/python/learn/learn_io/generator_io.py:26: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
    from collections import Container

tests_tf/test_attacks.py::TestSaliencyMapMethod::test_generate_np_targeted_gives_adversarial_example
  cleverhans/cleverhans/attacks_tf.py:27: UserWarning: attacks_tf is deprecated and will be removed on 2019-07-18 or after. Code should import functions from their new locations directly.
    warnings.warn("attacks_tf is deprecated and will be removed on 2019-07-18"

tests_tf/test_attacks.py::TestSaliencyMapMethod::test_generate_np_targeted_gives_adversarial_example
  python3.7/site-packages/tensorflow_core/python/ops/variable_scope.py:452: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
    shape is not None and isinstance(shape, collections_lib.Sequence) and

tests_tf/test_attacks.py::TestSaliencyMapMethod::test_generate_np_targeted_gives_adversarial_example
  cleverhans/cleverhans/attacks/attack.py:249: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
    for value in fixed.values()):

tests_tf/test_attacks.py::TestSaliencyMapMethod::test_generate_np_targeted_gives_adversarial_example
  cleverhans/cleverhans/attacks/saliency_map_method.py:155: UserWarning: Downcasting labels---this should be harmless unless they are smoothed
    warnings.warn("Downcasting labels---this should be harmless unless"

tests_tf/test_attacks.py::TestSaliencyMapMethod::test_generate_np_targeted_gives_adversarial_example
  cleverhans/cleverhans/compat.py:22: UserWarning: <function reduce_sum_v1 at 0x7fcda46874d0> is deprecated. Switch to calling the equivalent function in tensorflow.  This function was originally needed as a compatibility layer for old versions of tensorflow,  but support for those versions has now been dropped.
    warnings.warn(str(f) + " is deprecated. Switch to calling the equivalent function in tensorflow. "

tests_tf/test_attacks.py::TestSaliencyMapMethod::test_generate_np_targeted_gives_adversarial_example
  cleverhans/cleverhans/compat.py:22: UserWarning: <function reduce_max_v1 at 0x7fcda468a710> is deprecated. Switch to calling the equivalent function in tensorflow.  This function was originally needed as a compatibility layer for old versions of tensorflow,  but support for those versions has now been dropped.
    warnings.warn(str(f) + " is deprecated. Switch to calling the equivalent function in tensorflow. "

tests_tf/test_attacks.py::TestSaliencyMapMethod::test_generate_np_targeted_gives_adversarial_example
  cleverhans/cleverhans/compat.py:22: UserWarning: <function reduce_any_v1 at 0x7fcda468add0> is deprecated. Switch to calling the equivalent function in tensorflow.  This function was originally needed as a compatibility layer for old versions of tensorflow,  but support for those versions has now been dropped.
    warnings.warn(str(f) + " is deprecated. Switch to calling the equivalent function in tensorflow. "

-- Docs: https://docs.pytest.org/en/latest/warnings.html
======================================================================================== 1 failed, 9 warnings in 13.54s =========================================================================================

This can be reproduced by using these seeds in the setUp function:

 def setUp(self):  
    tf.set_random_seed(2825845284096387624)
    super(TestSaliencyMapMethod, self).setUp()

    self.sess = tf.Session()
    self.sess.as_default()
    self.model = DummyModel()
    self.attack = SaliencyMapMethod(self.model, sess=self.sess)

The test fails one in 30 times that I tried. Is there a way to fix this? Should the assertion be adjusted to be >=0.9?

loopylangur commented 4 years ago

cc @ftramer