search

raghakot / keras-vis

Neural network visualization toolkit for keras
https://raghakot.github.io/keras-vis
MIT License
2.97k stars 664 forks source link

Fix #102 #130

Closed soroushj closed 5 years ago

soroushj commented 5 years ago

The grad-CAM example did not work, because of jet_heatmap having an incorrect shape. This PR fixes the issue.

keisen commented 5 years ago

Thank you for PR.

102 was not reproduce in my test environments.

Can you post the short script (or the link of Gist) here that can reproduce #102 ?

soroushj commented 5 years ago

Thank you for your review, @keisen.

I reproduced #102 by running the notebook cells in order.

My environment config is as follows:

Running the grad-CAM cell leads to the following error:

ValueError                                Traceback (most recent call last)
<ipython-input-6-9a4167e00119> in <module>()
     17         print('np.uint8(cm.jet(grads)[..., :3] * 255)', np.uint8(cm.jet(grads)[..., :3] * 255).shape)
     18         jet_heatmap = np.uint8(cm.jet(grads)[..., :3] * 255)
---> 19         ax[i].imshow(overlay(jet_heatmap, img))

~/.local/lib/python3.5/site-packages/vis/visualization/__init__.py in overlay(array1, array2, alpha)
     46         raise ValueError("`alpha` needs to be between [0, 1]")
     47     if array1.shape != array2.shape:
---> 48         raise ValueError('`array1` and `array2` must have the same shapes')
     49 
     50     return (array1 * alpha + array2 * (1. - alpha)).astype(array1.dtype)

ValueError: `array1` and `array2` must have the same shapes

The cause is that jet_heatmap has an incorrect shape of (224, 224, 3, 3). Note that cm.jet(grads) returns an array of shape (224, 224, 3, 4), while img and grads have a shape of (224, 224, 3).

Using the cmap argument of imshow instead of cm.jet solves the problem. This is also how other examples in the notebook handle color-mapping. I've tested it and it correctly produces the same images shown in the notebook.

P.S.: A 'jet' color-mapping is not necessary at all. It could be as simple as imshow(overlay(grads, img)).

keisen commented 5 years ago

Thank you for your detail report. I’m not sure what’s the cause of problem and the difference of our environments.

I tried that I modified examples/vggnet/attention.ipynb like your patch and it ran cells in order. The result is following.

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-7-74c11d7e744b> in <module>()
     14         # jet_heatmap = np.uint8(cm.jet(grads)[..., :3] * 255)
     15         # ax[i].imshow(overlay(jet_heatmap, img))
---> 16         ax[i].imshow(overlay(grads, img), cmap='jet')

~/workspace/keras-vis/vis/visualization/__init__.py in overlay(array1, array2, alpha)
     46         raise ValueError("`alpha` needs to be between [0, 1]")
     47     if array1.shape != array2.shape:
---> 48         raise ValueError('`array1` and `array2` must have the same shapes')
     49 
     50     return (array1 * alpha + array2 * (1. - alpha)).astype(array1.dtype)

ValueError: `array1` and `array2` must have the same shapes

When it don’t modify, this error is not occurred.

Please make sure your environments one more, and give me following informations.

soroushj commented 5 years ago

It could be that the behavior of matplotlib.cm.jet and/or vis.visualization.visualize_cam is different on our environments.

For me, in the original example, the shape of arrays will be as follows:

Can you check the shape of these arrays on your machine?

I'm using:

keisen commented 5 years ago

Can you check the shape of these arrays on your machine?

I got following results. This is as expected.

  • grads: (224, 224, 3)

This is explicitly incorrect shape...

keras-vis 0.4.1 with vis/optimizer.py at master -- see #120

Is this means what you modified vis/optimizer.py on keras-vis 0.4.1 (i.e., release-0.4.1 )? If so, the keras-vis in your environment may have any bugs because its version has many problems. Please retry to install keras-vis as follows.

pip install git+https://github.com/raghakot/keras-vis.git
soroushj commented 5 years ago

I installed keras-vis from the GitHub repo and the issue did not recur. Thank you, @keisen.