twosixlabs / armory

ARMORY Adversarial Robustness Evaluation Test Bed
MIT License
176 stars 67 forks source link

store output and experiment for reruns #1801

Open mwartell opened 1 year ago

mwartell commented 1 year ago

It would be swell to allow armory to rerun the output file and compare new results to old.

In current parlance it would look like

$ armory run mnist_baseline.json
 # creates an output.json containing mnist_baseline.json and a `results` block

You should be able to

$ armory run output.json

which compares the old results to the new.

related to #1310

mwartell commented 1 year ago

Related to #1727