`collect-outputs` util

[jprokos26]

Can be utilized to generate evaluation tables automatically utilizing various flags. Organize by performer, attack type, or any other config field. Additionally clean up your output directory automatically.

• Remove folders with just armory-log.txt and colored-log.txt
• Remove pairs of duplicate png images from saved_samples
• Removed files are placed in ~/.armory/outputs/.cleaned

Example: After running:

cd ~/git/twosixlabs/gard-evaluatsion/MITM_sleeper_agent/jprokos26
python3 -m armory utils collect-outputs --collate --glob "*Poison*.json" --sort --filter "diffusion"
cp -Lr ~/.armory/results/SleeperAgentAttack* .
git commit -am "Adding diffusion sleeper results"

Note, cwd affects href link for config paths See https://github.com/twosixlabs/gard-evaluations/tree/eval7-jp-MITM_sleeper/MITM_sleeper_agent/jprokos26 for output

[jprokos26]

Need to add flag to change where it is looking for results files instead of just ARMORY_OUTPUT_DIR

[jprokos26]

Addressed by https://github.com/twosixlabs/armory/pull/1986/commits/723c810c601b72f36d4db2bc5894b8f53692dcc0.

Stale: Add runtime field. Can be extracted from armory-log.txt

for file in SleeperAgentAttack/*/armory-log.txt; do echo -n $(basename $(find $(dirname $file) -name "*.json")); echo -n " - "; tail -n 1 $file | cut -d" " -f3; done

[jprokos26]

If defense is null, should grab other kwargs https://github.com/twosixlabs/armory/blob/master/scenario_configs/eval6/poisoning/sleeper_agent/baseline_defenses/cifar10_sleeper_agent_p10_perfect_filter.json#L58-L64