twosixlabs / armory

ARMORY Adversarial Robustness Evaluation Test Bed
MIT License
176 stars 67 forks source link

Have --check runs modify attack parameters #1756

Closed lcadalzo closed 2 years ago

lcadalzo commented 2 years ago

E.g. max_iter

davidslater commented 2 years ago

Poisoning attacks may have additional parameters to modify, like config["adhoc"]["train_epochs"] and "max_epochs" and "model_retraining_epoch" in sleeper agent.

Imperceptible_asr also has max_iter_1 and max_iter_2.

I think we already correctly handle nb_epochs in fit_kwargs.

lcadalzo commented 2 years ago

One thing to consider is that if a config doesn't specify a certain param (thus using the attack's default value), the logic of checking for the param in the config won't suffice. We could get around this by using getattr() after the attack is loaded, but given it's pretty uncommon to leave a param like max_iter unspecified in the config, I think I prefer adding new --check logic to the part of the code where it already lives (i.e. at scenario construction)

davidslater commented 2 years ago

Let's just document what we're doing, and not try to be extra fancy with getattr(). Wrapped functions would be particularly hard to deal with there, anyhow.