visualize_cam with multiple inputs

[juliojj]

Is it possible to use visualize_cam with a network having multiple inputs? I have a modified version of resnet50, with 3 inputs (2 are images and 1 is audio). I want to visualize activations of input1 or input2 (which are images) as in this example: https://github.com/raghakot/keras-vis/blob/master/examples/resnet/attention.ipynb However, if I give as input to visualize_cam "seed_input=input_1" or "seed_input=[input_1, input_2, input_3]", I get the error. Any help is more than appreciated. Thanks

grads = visualize_cam(model, layer_idx_m1, filter_indices=None,seed_input=scene, penultimate_layer_idx=penultimate_layer_m1) File "/usr/local/lib/python2.7/dist-packages/vis/visualization/saliency.py", line 239, in visualize_cam return visualize_cam_with_losses(model.input, losses, seed_input, penultimate_layer, grad_modifier) File "/usr/local/lib/python2.7/dist-packages/vis/visualization/saliency.py", line 160, in visualize_cam_withlosses , grads, penultimate_output_value = opt.minimize(seed_input, max_iter=1, grad_modifier=grad_modifier, verbose=False) File "/usr/local/lib/python2.7/dist-packages/vis/optimizer.py", line 122, in minimize seed_input = self._get_seed_input(seed_input) File "/usr/local/lib/python2.7/dist-packages/vis/optimizer.py", line 85, in _get_seed_input desired_shape = (1, ) + K.int_shape(self.input_tensor)[1:] File "/usr/local/lib/python2.7/dist-packages/keras/backend/tensorflow_backend.py", line 584, in int_shape return tuple(x.get_shape().as_list()) AttributeError: 'list' object has no attribute 'get_shape'

[illith]

i also face the same problem... have you solved it?

[juliojj]

I adapted this code: http://ethen8181.github.io/machine-learning/keras/resnet_cam/resnet_cam.html to receive multiple-inputs and to generate the CAM of a particular input. However, I am not sure if I did something wrong or if the difficulty of explaining the results are due to the complexity of my model, the given inputs, or the task (i.e., regression).

[illith]

@juliojj thank you!!!! I also found a good code: https://github.com/philipperemy/keract

[keisen]

Hi, there. Thank you for your report. We too want to visualize with multiple inputs, but we've not been working for it yet, sorry.

p.s., Instead, I needed it for my own experiments, so I developed below library. Would you please try it if it's okay.

https://github.com/keisen/tf-keras-vis

Thanks!

[gabarlacchi]

Hi to all, I have the following network in order to classify video sequences composed bu 30 frames. `video = Input(shape=(30,299,299,3)) inc = InceptionV3( weights='imagenet', include_top=False, input_shape=(299, 299, 3) ) cnn_out = GlobalAveragePooling2D()(inc.output) cnn = Model(inputs=inc.input, outputs=cnn_out) encoded_frames = TimeDistributed(cnn)(video) encoded_sequence = LSTM(128, activation='relu', return_sequences=False, kernel_initializer=he_uniform(), bias_initializer='zeros', dropout=0.5)(encoded_frames) hidden_layer = Dense(1024, activation='relu', kernel_initializer=he_uniform(), bias_initializer='zeros')(encoded_sequence) outputs = Dense(4, activation="softmax", kernel_initializer=glorot_normal(), bias_initializer='zeros')(hidden_layer) model = Model(inputs=[video], outputs=outputs)

adam = Adam(lr=0.001, beta_1=0.9, beta_2=0.999, amsgrad=False) model.compile(optimizer=adam, loss='categorical_crossentropy', metrics=['accuracy']) model.summary()`

Do you have idea how can I visualize the activations of each image fed in input? It should be related to the class that the final dense layer has predicted? Do you think it can works?

Any help is really appreciated, thanks!