Error in Class Activation map for hidden layers

[gprabaha]

I was trying out the class activation map of some cat and dog images for different layers of VGG16 (using keras.applications.vgg16, not vis.utils.vggnet), and the code, shown in (https://raghakot.github.io/keras-vis/visualizations/attention/) does not seem to be working for layers other than 'predictions'. When I use layer_name='block1_conv2', it is throwing an error, ValueError: Unable to detect penultimate 'Convolution2D' or 'Pooling2D' layer for layer_idx: 2. On the other hand, for layer_name='block2_conv2', or layer_name='block3_conv2' there is an AssertionError for len(item_slice) == 4.

Further, when I am using layer_name='predictions', the prediction is performed correctly for the cat and dog image (Tabby, and Siberian Husky), but the heat map is barely on the dog.

The problem is seen even for the images shown in the website, at least for the Tiger and the Spider Web.

When I use the VGG16 from vis.utils.vggnet, though, the heatmap makes sense!

Though, even in this case, when I change the layer to fc1 or fc2, the entirety of the image lights up.

And the same error as mentioned before shows up when I try see the map for intermediate Convolution layers. How can this be solved?

The code I am using is:

import numpy as np
import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt
import matplotlib.axis as ax

from keras.preprocessing.image import img_to_array
from keras.applications.imagenet_utils import preprocess_input

from vis.utils import utils
from keras.applications.vgg16 import VGG16, decode_predictions
from vis.visualization import visualize_cam

# Build the VGG16 network with ImageNet weights
model = VGG16(weights='imagenet', include_top=True)
print('Model loaded.')

# The name of the layer we want to visualize
# (see model definition in vggnet.py)
layer_name = 'predictions' # 'block1_conv2', 'block2_conv2', 'block3_conv2'
layer_idx = [idx for idx, layer in enumerate(model.layers) if layer.name == layer_name][0]

# Images corresponding to cat, and dog
image_paths = [
    "cat.jpg",
    "dog.jpg"
]

heatmaps = []
for path in image_paths:
    seed_img = utils.load_img(path, target_size=(224, 224))
    x = np.expand_dims(img_to_array(seed_img), axis=0)
    x = preprocess_input(x)
    pred_class = np.argmax(model.predict(x))
    print('Predicted class is ' +str(decode_predictions(model.predict(x), top=1)[0][0]))
    # Here we are asking it to show attention such that prob of `pred_class` is maximized.
    heatmap = visualize_cam(model, layer_idx, [pred_class], seed_img)
    heatmaps.append(heatmap)

plt.imshow(utils.stitch_images(heatmaps))
plt.axis('off')
plt.title('Class Activation map')
plt.savefig('keras-vis-cam-2.jpeg')

[wendywangwwt]

I got similar constant heatmap (but mine are covered by a blue layer). #27 mentioned this problem as well and I think it's not fully fixed?

[raghakot]

Can you guys try this again in latest? There were a couple of big changes with the latest commit.

[wendywangwwt]

@raghakot I'm using the latest version on PyPI (0.3.1).

from keras.preprocessing.image import ImageDataGenerator
from keras.models import load_model

batch_size = 2
test_datagen = ImageDataGenerator(rescale=1./255)
validation_generator = test_datagen.flow_from_directory(
        'validation',
        target_size=(200, 200),
        batch_size=batch_size,
        class_mode='binary',
        shuffle=False)

model = load_model('weights.hdf5')
res_cnn = 1- best.predict_generator(validation_generator,steps=1)

res_cnn_class = [0 if x < 0.5 else 1 for x in res_cnn[:,0]]

import numpy as np
import matplotlib
import matplotlib.pyplot as plt

from vis.utils import utils
from vis.visualization import visualize_cam

model = model

fig = plt.figure(figsize=(20,20))
i = -1
for fn in validation_generator.filenames:
    i += 1
    subplot = fig.add_subplot(1,12,i+1)

    if i < 1:
            subplot.set_title('Class A')
    else:
        subplot.set_title('Class B')

    seed_img = utils.load_img('validation/'+fn, target_size=(200,200))

    heatmap = visualize_cam(best, 7, [res_cnn_class[i]], seed_img)
    plt.imshow(heatmap, cmap=plt.get_cmap('gray'))
    plt.axis('off')

plt.show()

Attached zip file contains model weights, python notebook of the code, and test images to reproduce the problem. debug.zip

[gprabaha]

@raghakot I reinstalled keras-vis from source, and the new error that is now showing up is,

Traceback (most recent call last):
  File "keras-vis-cam.py", line 38, in <module>
    heatmap = visualize_cam(model, layer_idx, [pred_class], seed_img)
  File "/home/yx96/anaconda2/lib/python2.7/site-packages/keras_vis-0.3.1-py2.7.egg/vis/visualization.py", line 371, in v
isualize_cam
    penultimate_output = penultimate_layer.output
AttributeError: 'numpy.ndarray' object has no attribute 'output'

It seems to be failing to get the output from the penultimate layer! This error is showing, of course, for all acceptable layer_name, like, 'predictions', or 'block2_conv2'.

[raghakot]

Hi, The API has changed. There is now visualize_class_cam and visualize_regression_cam with visualize_cam being more general that works on custom loss functions. You might want to use visualize_class_cam now. API docs should be updated. I am still working on completely overhauling the docs and examples.

[wendywangwwt]

@raghakot Thanks for the update. After updating the package to 0.3.2 I'm able to import the new function visualize_class_cam. However now the output are pure blue squares (I just simply replaced visualize_cam with visualize_class_cam) :

[raghakot]

It is very likely the case that the penultimate_layer has a spatial resolution of (1, 1). You might have to specify a pre Conv layer manually via penultimate_layer_idx. I added examples for MNIST in examples/. Highly recommend checking that out to understand grad-CAM.

Is your code the same as this?

[wendywangwwt]

@raghakot The MNIST example runs well on my machine but my model still gets these blue squares even after I set the penultimate layer idx (now the idx corresponds to a 48x48x64 maxpooling layer) for CAM. visualize_class_saliency also gives me these blue squares. My input size is 200x200 and the output size of last maxpooling layer is 48x48x64. The code for both CAM and saliency map I used is included in a notebook in this compressed file, along with two test images and the model weights. debug.zip

[raghakot]

@wanting0wang It appears that the gradients are all 0's with respect to the input. This could mean that your model is overfitting or not trained properly since none of the input pixels have an effect on the output. How was this trained? It also seems too shallow. You probably need more conv layers and reduce spatial resolution down to (7, 7) or (5, 5) before applying Dense layer.

Also, this is unrelated to MNIST dataset, so you should create a separate issue for it and avoid cluttering this thread. I will help with debugging on the new thread.

[raghakot]

I am closing this due to lack of correspondence. Feel free to reopen the issue if you're still encountering problems.

[wendywangwwt]

@raghakot Hi sorry for the late reply.. I was on holiday. I think my problem is because of the high spatial resolution (and that increases the parameters a lot and leads to overfit). I trained another model with lower spatial resolution in the last maxpooling layer (about 9x9) and everything looks great. Thank you for the suggestions!

[raghakot]

Great. Glad it worked out :)