search

twosixlabs / armory

ARMORY Adversarial Robustness Evaluation Test Bed
MIT License
176 stars 67 forks source link

1982 export hotfix #1983

Closed jprokos26 closed 1 year ago

jprokos26 commented 1 year ago

See #1982

jprokos26 commented 1 year ago

@swsuggs I couldn't find documentation about what the model is supposed to output but when testing with other configs they all output numpy arrays; do we notate what the model output is required to be anywhere? Is there supposed to be a postprocessing function to handle this?

swsuggs commented 1 year ago

@jprokos26 I think Armory's use of numpy is mainly to conform with ART. I found this in the doc:

Currently, the ART toolbox only accepts NumPy arrays as inputs to attacks, defenses and model fitting. It is understood that this is an inefficient way to utilize data since it requires a conversion from FrameworkTensor -> ndarray -> FrameworkTensor.

Which doesn't mention model output, but I would assume it applies to output as well. I haven't dug into the ART documentation though. This is a performer model that's outputting a tensor? Perhaps they are doing some post-processing within the "model" but outside of the art wrapper that converts it back to a tensor? If so, I think it would be better to edit their code (in gard-submissions) to return numpy rather than making armory conform to their submission.

jprokos26 commented 1 year ago

@swsuggs Got it, I suspected this was a performer issue, I just was not sure where the post-processing occurred. I'll make that change in gard-submissions and close this. In the meantime anyone experiencing this issue can use jprokos26:1982-export-hotfix.