search

cleverhans-lab / cleverhans

An adversarial example library for constructing attacks, building defenses, and benchmarking both
MIT License
6.2k stars 1.39k forks source link

(Potentially) Faulty test cases? #1161

Closed iamgroot42 closed 3 years ago

iamgroot42 commented 4 years ago

The issue tracker should only be used to report bugs or feature requests. If you are looking for support from other library users, please ask a question on StackOverflow.

Describe the bug It seems some of the tests for attacks include include unintentional bugs? To be specific, some targeted attacks sample targets as np.random.randint(0, 1, n_samples). However, this function takes generates integers in [lower_range, upper_range) and thus, in this case, will always give 0. Is this intentional behaviour? (Example here)

Expected behavior Ideally the attack should work well in both scenarios, but using random samples may be a better test (instead of a model that would be biased towards one of the labels and pass some sub-tests?)

jonasguan commented 3 years ago

Hey @iamgroot42 , you have a very keen eye! This does look like a bug. However, we are discontinuing support for TF1, and will soon mark the current version of CleverHans as the final v3 release, before we update to v4. In an abundance of caution, I won't make the change since this is part of the test of the older TF1 code, lest I somehow break it right before release. However, if you notice this bug in the newer v4 code (changes upcoming!), please let us know in a new issue or PR. I know we haven't been active for a while, but we're back now :)