search

twosixlabs / armory

ARMORY Adversarial Robustness Evaluation Test Bed
MIT License
176 stars 67 forks source link

Add support for ablation studies. #1929

Open jprokos26 opened 1 year ago

jprokos26 commented 1 year ago

I.e. take in a collection of config files (or optimally a set of learning-rates or base-images or just a generic list of kwargs to modify) and output a single result file with each in a separate "result" object.

swsuggs commented 1 year ago

Are you aware of Armory's sweep attack capability? It's not the exact same thing, but maybe partially meets this need. It sounds like you are looking for a full result for each parameter value, while the sweep attack goes from weakest to strongest and I believe it skips samples if they were successfully broken with weaker parameters.