dependable-ai / nn-dependability-kit

Toolbox for software dependability engineering of artificial neural networks
GNU Affero General Public License v3.0
34 stars 9 forks source link

Allow perturbation metric to have a separate function for evaluation of performance drop #13

Open chengchihhong opened 6 years ago

chengchihhong commented 6 years ago

Currently, the perturbation loss metric assumes that the result is based on classification, so the performance drop is just the difference of output of softmax for a specific class, before and after perturbation.

To use it in other non-classification tasks, one needs to define what does it mean to be performance drop. This should be refactored as a separate function to be injected during the computation of the performance loss.

The default should be classification.

chengchihhong commented 5 years ago

In addition, it can be generalized not only to performance drop, but as an indicator on result deviation.