search

marcoancona / DeepExplain

A unified framework of perturbation and gradient-based attribution methods for Deep Neural Networks interpretability. DeepExplain also includes support for Shapley Values sampling. (ICLR 2018)
https://arxiv.org/abs/1711.06104
MIT License
725 stars 133 forks source link

How to get the heatmap instead of overlay image? #12

Closed kilickaya closed 6 years ago

kilickaya commented 6 years ago

The method works perfectly fine on a new dataset with another model I trained on. However from what I understand .explain() method returns a (H,W,3) sized gradient image where reddish values correspond to positive evidence and blueish values to negative.

I wonder how can one directly get the heatmap, instead an overlay image of edge map on top of the heatmap?

marcoancona commented 6 years ago

Hi. You can look in the utils file how to display the heatmap. You can simply use Matplotlib's imshow for to get the heatmap from the numpy array. Does this answer your question?

kilickaya commented 6 years ago

Thanks for the answer!

I think I should make it more clear. Given an input image of (H,W, 3), I want to have a map for each pixel (H,W) where values close to 1 indicates positive relevance. However, the result returned by, let's say de.explain('saliency', logits, X, xs) is a 3-channel map, which is good for visualization but not for quantitative analysis:

Let's say I want to sort pixels in terms of their positive importance value, so that further I want to perturb the image and observe the accuracy drop (if any). In a way, I want to replicate Figure 3 of https://openreview.net/pdf?id=Sy21R9JAW

How should I proceed for this?

marcoancona commented 6 years ago

I see. You get a 3-channel attribution map because your input has also 3 channels. In other words, you get an attribution score for each input pixels and each channel. Now you can decide how to get from here to a single channel heatmap. What is done in the related literature (and also in our paper) is to sum up along the channel dimension (average is also possible). Notice that what de.explain() returns is not an RGB image. Is a saliency score for each input channel.

kilickaya commented 6 years ago

Summing across channels makes sense to me. Will try to reproduce results for sensitivity-n analysis. Thanks a lot for that!

Subh1m commented 2 years ago

Where can I find the sensitivity-n evaluation source code?

Subh1m commented 2 years ago

I found the sensitivity-n code in the shapely branch. It seems we are creating a deltas.hdf5 file for the sensitivity-n test. Can you please explain how to understand the results? I extracted the data from the hdf5 file and got an array for each attribution method.