Error when running attention.ipynb

[taylorty]

Hi,

I tried to run attention.ipynb here with Tensorflow backend. But it showed an error in the last section of the code saying that "ValueError: array1 and array2 must have the same shapes"

ValueError Traceback (most recent call last)

in () 15 print(jet_heatmap.shape) 16 print(img.shape) ---> 17 ax[i].imshow(overlay(jet_heatmap, img)) /Users/xxx/anaconda/lib/python2.7/site-packages/vis/visualization/__init__.pyc in overlay(array1, array2, alpha) 46 raise ValueError("`alpha` needs to be between [0, 1]") 47 if array1.shape != array2.shape: ---> 48 raise ValueError('`array1` and `array2` must have the same shapes') 49 50 return (array1 * alpha + array2 * (1. - alpha)).astype(array1.dtype) ValueError: `array1` and `array2` must have the same shapes I've checked the versions of Tensorflow, Keras and keras-vis lib and they are all updated.

[linchundan88]

I got the same error, too.

[linchundan88]

jet_heatmap = np.uint8(cm.jet(grads)[..., :3] * 255) ax[i].imshow(overlay(jet_heatmap, img)) #this line raise an error

jet_heatmap shape (224,224,3,3), img shape(224,224,3)

[linchundan88]

jet_heatmap = np.uint8(cm.jet(grads)[..., :3] * 255)

I changed the previous line to the following line . jet_heatmap = np.uint8(cm.jet(grads) * 255)[:, : , :, 0]

Maybe it is due to different python version's slice method is different?

[linchundan88]

Anyway, this project is very helpful.

[dietercastel]

I'm having the same error running it from a regular python script based on the ipython notebook. I'm using python3, keras 2.0.8 and matplotlib 2.0.2. In the matplotlib in their change log they mention some changes to colormaps. But I can't see why it would be breaking changes although I'm not familiar with the matplotlib internals.

The weird thing for me is the 4th dimension the function cm.jet returns. cm.jet(grads).shape = (299,299,3,4) I can understand that it would have shape (w,h,4) where the third dimension would be a RGBA vector of length 4. In that case @raghakot 's code [...,:3] would just remove the alpha channel, the 4th element of that vector, across the tensor. Which makes sense if i look at the overlay function (since there he's mixing the alphas himself).

Right now as a workaround I'm just displaying the grads side by side with the images. Using ax[i][1].imshow(grads, cmap='jet'). Although weirdly i seem to be having inverse coloring as in the ipython notebook. Full code here.

@linchundan88 @taylorty What versions are you guys running? Matplotlib 2.0.2 also?(check with pip list or conda list)

[fujikosu]

ax[i].imshow(overlay(grads, img)) Instead of ax[i].imshow(overlay(jet_heatmap, img)) Worked fine for me

[cklat]

Any news to this issue? None of the workarounds mentioned here, give me the same heatmap for the Grad-Cam Visualization as in the attention.ipynb tutorial.

Weirdly, I got the full code working on a second machine after several attempts of cloning, updating und erasing the repository on that machine but I couldn't get it to work on my primary machine. The machine, on which the code is working straight on, is using matplotlib 2.0.2 and keras-vis 0.4.1

[phisad]

The problem seems to be the matplotlib, since cm.jet(...) is returning 4 jet images. So I just overlayed each individually. The 4th returning image seems to be the alpha channel and is mostly only white. Additionally, I ignored the normalization step for uint 255. This gave me some reasonable results.

       # Lets overlay the heatmap onto original image.    
        jet_grads = cm.jet(grads)
        overlay_img = img
        for idx in [0, 1, 2]:
            jet_heatmap = jet_grads[..., idx]
            overlay_img = overlay(overlay_img, jet_heatmap)
        ax[i].imshow(overlay_img)

[mimoralea]

@phisad can you expand a little more how you solved it?

I'm stuck on visualize_attention.ipynb, and I think the issue is coming from the file vis/visualization/saliency.py. The heatmaps there, in the function visualize_saliency_with_losses and visualize_cam_with_losses, are returning as a black and white image, no color channels. So, overlaying that "black and white" heatmap on top of the color images is causing the issue (at least for me - with unmodified code).

[phisad]

@mimoralea OK, the closest I can get to the visualize_attention.ipynb narrative is to just select the first from the cm.jet(grads).

        # Lets overlay the heatmap onto original image.    
        jet_heatmap = cm.jet(grads)[..., 0]
        ax[i].imshow(overlay(jet_heatmap, img))

The uint conversion is depending on your image scale (if [0...255] or [0...1]).

There might be a change in matplotlib behavior that introduced the problem. In the commit from fix #67 master@raghakot raghakot committed on Aug 24, 2017 there was actually before return np.uint8(cm.jet(grads)[..., :3] * 255)[0] We could interpret [0] as "selecting the first returning map".

But the changed notebook states only np.uint8(cm.jet(grads)[..., :3] * 255) without [0].

[mimoralea]

I see. FWIW, I think my issue is slightly different. It has to do with my model being trained on grayscale images. So, when I try to overlay the heatmap (which is 1 channel as well), then the plot methods throws a similar error.

Thanks so much for the help @phisad!