twosixlabs / armory

ARMORY Adversarial Robustness Evaluation Test Bed
MIT License
176 stars 67 forks source link

Add support for pre-computed fairness majority masks #1964

Closed f4str closed 1 year ago

f4str commented 1 year ago

Modify the poisoning scenario to accept the majority_masks field which will attempt to read a pre-computed npz file which contains the majority masks for the train and test data. This will override the explanatory model if provided.

The npz file must be placed in the root of the armory directory (under armory/). So the provided path:

"adhoc": {
    ...
    "compute_fairness_metrics": true,
    "majority_masks": "audio_majority_masks.npz",
    ...
}

will look for armory/audio_majority_masks.npz. The file will have two keyword arg arrays: train and test which are boolean arrays the same length as the train and test sets, respectively.

This works on both docker and non-docker mode.

f4str commented 1 year ago

A zip file containing the majority masks for the speech commands dataset and a sample config file for testing: majority_masks.zip

swsuggs commented 1 year ago

Thank you Farhan, this is great. I'm just adding the npz file to its own directory with a bit of extra code to facilitate loading it.