I am transferring a discussion between Ying, Dr. Jamshidi, and me from email to benefit anyone who might have the same confusion.

Me: "In https://github.com/csce585-mlsystems/project-athena/blob/grade-2020/Task2/Epsilon/Comments.md, you write "Experimental setting. What are the weak defenses in the target ensemble --- the ensemble you used to generated the AEs?" In https://github.com/smithandrewk/project-athena/blob/master/Task2/report_task2.ipynb, we write in the evaluation section, "73 weak defenses consisting of a clean model followed by 72 models trained on transformed input data. The full configuration of this model can be located at "./Task2/configs/athena-mnist.json"." Could you specify what you meant by this feedback?"

Ying: "For comment 1. I doubted that the AEs were generated targeted on an ensemble of 73 weak defenses due to the computational cost. For example, it took me around 5 minutes to generate 5 AEs (PGD+EOT, 500 samples) targeted on an ensemble of 3 weak defenses on a computer of 8 core CPUs + 32 GB memory. It will take much longer to generate a single AE when targeting an ensemble of 73 weak defenses. So, I guess what happened was that you generated the AEs targeted on an ensemble of 3 weak defenses (same setting as the demo) and then later you updated your repo from the project repo (where I updated the configuration with all 73 models)."

Me: " I am significantly confident that we generated our adversarial examples using the weak defenses as we outlined in the report. For example, to generate a single AE (PGD-EOT, 500 samples) on my machine with 16 GB RAM and 8 core CPUs, the computation took approximately 45 minutes. Additionally, when we load the pool for the target model, we load the config file with 73 weak defenses." NOTE : I am incorrect in that I did not load the config with 73 weak defenses

Ying: "The first factor is the size of the dataset (i.e., the number of benign samples) for which you want to generate the AEs. This is defined by the size of "data_bs" in the script.

The second factor is the number of samples in the distribution when you are using EOT, which is defined by the "num_sample" in the json file for the attack (this is the 500 you mentioned). For non-EOT attacks, we do not have this parameter (for example, Task 1 assignment).

The third factor is the size of the target ensemble (i.e., the number of weak defenses in the ensemble), which is defined by the json file for the ensemble (athena-mnist.json in this case).

If you check the process to generate AEs in the tutorials for Task 2 Option 1, you will see that, to generate AEs, we need (1) a target model (we load the information of weak defenses, then weak defense models via load_pool(), and then create an ensemble model as the target model of the attack); (2) the images for which you want to generate AEs (therefore, we load the bs samples from the "npy" file we defined in the data json file. This is the number of benign samples I meant in the previous letter.); (3) the configuration of the attacks (therefore, we load the adversarial configurations from the json file in which we defined the values for tunable parameters. If you use EOT attack, there is a parameter named "num_sample", which is the second factor I mentioned above). Fed all this stuff, our AE generator will generate one AE per benign sample per attack variant, targeting the specific target model. For example, in the demo I provided (see the setting shown below ), I generated 5 FGSM-EOT AEs (computed the loss over the distribution of 1000 randomly rotated samples) and 5 PGD-EOT AEs (computed the loss over the distribution of 500 randomly translated samples), targeting an ensemble that consists of 3 weak defenses.

related script pieces: (1) PREPARE THE TARGET MODEL in "athena-mnist.json"

"num_transformations": 73,
"active_wds": [10, 20, 30],

in "craft_adversarial_examples.py", load weak defenses and create the target ensemble,

# In the context of the white-box threat model,
# we use the ensemble as adversary's target model.
# load weak defenses (in this example, load a tiny pool of 3 weak defenses)
pool, _ = load_pool(trans_configs=pool_configs,
                    model_configs=model_configs,
                    active_list=True,
                    wrap=True)
# create an AVEP ensemble as the target model
wds = list(pool.values())
target = Ensemble(classifiers=wds, strategy=ENSEMBLE_STRATEGY.AVEP.value)

with "active_list=True", load_pool() will return you the weak defenses specified by the "active_wds" in the "athena-mnist.json", so you do not have to update the whole file everytime."

Conclusion

Though in "athena-mnist.json" we had all 72 weak defenses and the 1 undefended model, the parameter "active_wds" did not contain all 72 weak defenses; thus, we only evaluated against weak defenses 10, 20, and 30, as specified in the configuration. I did not consider the "active_list" parameter in the "load_pool" method. Do not make this same mistake! Thank you Ying for your help.

csce585-mlsystems / project-athena

Weak defense confusion and "active_list" parameter #40

Conclusion