[question] Is there any established way of compering the results of two methods of model explaination?

[tugot17]

My question regards explanation methods for image classifiers. Is there established way to compere two different "explanations"? My idea was to e.g. use the segmentation masks (assuming we have one) and compere IoU value between the original mask and the part of an input image that model is "looking at". What do you think about such approach, is there anything similar already in captum?

[bilalsal]

Hi @tugot17, thank you for this proposal!

your proposal sounds similar to Network Dissection http://netdissect.csail.mit.edu/

The masks could serve as a sort of ground truth. However, the model might also rely on features outside of the segmentation mask to decide for / against a certain target. You can follow such behavior in these tutorials: https://captum.ai/tutorials/Resnet_TorchVision_Ablation https://captum.ai/tutorials/Segmentation_Interpret

There have been some quality metrics to evaluate attributions algorithms. Captum implements two of these metrics: infidelity and max-sensitivity. These metrics are described in the following NeurIPS'19 article by Yeh et al: https://papers.nips.cc/paper/9278-on-the-infidelity-and-sensitivity-of-explanations.pdf More on this in Hooker et al. article @NeurIPS'19 https://arxiv.org/pdf/1806.10758.pdf

We are actively evaluating further metrics and approaches to provide insights into and user control over computed attributoins.

Hope this helps

[tugot17]

Where in captum I can find the implementation of those methods (both infidelity and max-sensitivity doesn't return any search results when I use the search engine from the docs)?

[bilalsal]

Hey @tugot17

Here you can find the implementation of these metrics at https://github.com/pytorch/captum/tree/master/captum/metrics

In the next release, we will provide notes and a white paper describing these.

Hope this helps

[tugot17]

Yeah it does, I will check this out, thanks. For now I close the issue.