search

sicara / tf-explain

Interpretability Methods for tf.keras models with Tensorflow 2.x
https://tf-explain.readthedocs.io
MIT License
1.02k stars 112 forks source link

GradCam shows same activations for different class_index with use_guided_grads = True #146

Closed shaheen-syed closed 3 years ago

shaheen-syed commented 4 years ago

I have a trained 4 class CNN model based on a VGG16 architecture. I am using gradcam to inspect where in each image each class activates. However, when I use gradcam with use_guided_grads = True, I get the same heatmap for each class_index. I expect the activations to be different for class_index 0, 1, 2 and 3. When I set the use_guided_grads to False, then the output is as expected, that is, a different heatmap for each class. I am attaching two plots for illustrative purposes. Is this expected behaviour or am I missing something?

Activations of the same image for classes 0, 1, 2, 3 and use_guided_grads = True with_guided_grad

Activations of the same image for classes 0, 1, 2, 3 and use_guided_grads = False without_guided_grad

RaphaelMeudec commented 4 years ago

The guided gradients have a misleading effect on attribution map. It implies a heavy contribution of the raw input into the construction of the attribution map, and can cause to produce unexpected "valid" feature map for some classes.

A nice paper that is going though this effect (and others) is Sanity Check for Saliency Maps.

shaheen-syed commented 4 years ago

Thanks for letting me know. After I posted the issue I also come across the paper you mentioned, and some others addressing similar issues/concerns.

RaphaelMeudec commented 3 years ago

Closing this issue as it is more related to the method than the library.