New metrics: implement Consistency and Sufficiency

understandable-machine-intelligence-lab / Quantus

Quantus is an eXplainable AI toolkit for responsible evaluation of neural network explanations

https://quantus.readthedocs.io/

Other

547 stars 76 forks source link

New metrics: implement Consistency and Sufficiency #112

Closed annahedstroem closed 2 years ago

annahedstroem commented 2 years ago

Implement Consistency (that belongs to the Robustness category) and Sufficiency (that belongs to the Sufficiency category):

• Consistency: roughly, two instances x, x0 that get the same explanation should also have the same prediction. For instance, if two different images are assigned the same explanation, e(x) = e(x 0 ) = “contains a zebra”, then their assigned labels should also be the same. • Sufficiency: if x is assigned an explanation e(x) = π that also holds for another instance x 0 (even if e(x 0 ) 6= π), then x 0 should have the same label as x.

Paper: https://arxiv.org/pdf/2202.00734.pdf

annahedstroem commented 2 years ago

@dilyabareeva

annahedstroem commented 2 years ago

To clarify changes excepted in the PR, the should be reflected in the following folders:

[ ] quantus/metrics/faithfulness_metrics.py and quantus/metrics/robustness_metrics.py
[ ] tests/metrics/test_faithfulness_metrics.py and quantus/metrics/test_robustness_metrics.py
[ ] tests/metrics/test_faithfulness_metrics.py and quantus/metrics/test_robustness_metrics.py
[ ] and in the tutorial: https://github.com/understandable-machine-intelligence-lab/Quantus/blob/main/tutorials/tutorial_basic_example_all_metrics.ipynb
[ ] `tests/helpers/constants.py``
[ ] README.md

annahedstroem commented 2 years ago

Completed. https://github.com/understandable-machine-intelligence-lab/Quantus/issues/112