ResNet-50 visualization

[t-softic]

Hi,

I used fine-tuned resnet-50 and added a new classifier on top. When I try to visualize with CAM methods, I get meaningless heatmaps. On the other hand, when visualizing with gradient, there is an attention in regions that should be highlighted. I know ResNet is hard to visualize due to its skip connections, but do you have any tips on what I could do to improve CAM visualizations, or if there is any good library that could help me (I tried keras-vis same outcome).

Thanks, Tarik

[keisen]

Hi, @t-softic .

I believe that GradCAM method can work well with resnet-50 as follow:

https://github.com/raghakot/keras-vis/blob/master/examples/resnet/attention.ipynb

Could you please submit the heatmaps and a code snippet to reproduce and examine the problem?

Thanks!

[anotinelg]

Tensorflow 2.3.0

Hi, i have a "similar" problem.. it happens when i construct a new model from an already created.

Construction of model:

from tensorflow.keras.preprocessing.image import load_img
from tensorflow.keras.applications.resnet_v2 import ResNet50V2 as Model
from tensorflow.keras.applications.resnet_v2 import preprocess_input

# Load model
base_model = Model(weights='imagenet', include_top=False)

inputs = tf.keras.Input(shape=(224, 224, 3))
x = preprocess_input(inputs)
outputs = base_model(x)
model = tf.keras.Model(inputs=inputs, outputs=outputs)

result of the plot:

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_9 (InputLayer)         [(None, 224, 224, 3)]     0         
_________________________________________________________________
tf_op_layer_RealDiv_3 (Tenso [(None, 224, 224, 3)]     0         
_________________________________________________________________
tf_op_layer_Sub_3 (TensorFlo [(None, 224, 224, 3)]     0         
_________________________________________________________________
resnet50v2 (Functional)      (None, None, None, 2048)  23564800  
=================================================================
Total params: 23,564,800
Trainable params: 23,519,360
Non-trainable params: 45,440
_________________________________________________________________
None

then i basically code the same code which is in https://github.com/keisen/tf-keras-vis/blob/master/examples/attentions.ipynb

The error is:

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-13-6193c76b1152> in <module>
     67 cam = gradcam(loss,
     68               X,
---> 69               penultimate_layer=-1, # model.layers number
     70              )
     71 cam = normalize(cam)

~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tf_keras_vis/gradcam.py in __call__(self, loss, seed_input, penultimate_layer, seek_penultimate_conv_layer, activation_modifier, normalize_gradient, expand_cam, training)
     55         # Processing gradcam
     56         model = tf.keras.Model(inputs=self.model.inputs,
---> 57                                outputs=self.model.outputs + [penultimate_output_tensor])
     58         with tf.GradientTape(watch_accessed_variables=False) as tape:
     59             tape.watch(seed_inputs)

~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py in __new__(cls, *args, **kwargs)
    240       # Functional model
    241       from tensorflow.python.keras.engine import functional  # pylint: disable=g-import-not-at-top
--> 242       return functional.Functional(*args, **kwargs)
    243     else:
    244       return super(Model, cls).__new__(cls, *args, **kwargs)

~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow/python/training/tracking/base.py in _method_wrapper(self, *args, **kwargs)
    455     self._self_setattr_tracking = False  # pylint: disable=protected-access
    456     try:
--> 457       result = method(self, *args, **kwargs)
    458     finally:
    459       self._self_setattr_tracking = previous_value  # pylint: disable=protected-access

~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow/python/keras/engine/functional.py in __init__(self, inputs, outputs, name, trainable)
    113     #     'arguments during initialization. Got an unexpected argument:')
    114     super(Functional, self).__init__(name=name, trainable=trainable)
--> 115     self._init_graph_network(inputs, outputs)
    116 
    117   @trackable.no_automatic_dependency_tracking

~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow/python/training/tracking/base.py in _method_wrapper(self, *args, **kwargs)
    455     self._self_setattr_tracking = False  # pylint: disable=protected-access
    456     try:
--> 457       result = method(self, *args, **kwargs)
    458     finally:
    459       self._self_setattr_tracking = previous_value  # pylint: disable=protected-access

~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow/python/keras/engine/functional.py in _init_graph_network(self, inputs, outputs)
    189     # Keep track of the network's nodes and layers.
    190     nodes, nodes_by_depth, layers, _ = _map_graph_network(
--> 191         self.inputs, self.outputs)
    192     self._network_nodes = nodes
    193     self._nodes_by_depth = nodes_by_depth

~/anaconda3/envs/tensorflow2_p36/lib/python3.6/site-packages/tensorflow/python/keras/engine/functional.py in _map_graph_network(inputs, outputs)
    929                              'The following previous layers '
    930                              'were accessed without issue: ' +
--> 931                              str(layers_with_complete_input))
    932         for x in nest.flatten(node.outputs):
    933           computable_tensors.add(id(x))

ValueError: Graph disconnected: cannot obtain value for tensor Tensor("input_8_1:0", shape=(None, None, None, 3), dtype=float32) at layer "conv1_pad". The following previous layers were accessed without issue: []

[anotinelg]

hi @t-softic, have you tried to use GradCam on a Custom model that includes a resnet and then some top classifiers layers that you add?