multiple inputs CNN based regression problem

linbingru commented 3 years ago

Hi,

The document says that the tf-keras-vis are now supporting the muti-inputs model (two 2D matrices). However, with the code below occurred the error. Thanks for your helping!!

Best,

Create Gradcam object
gradcam = Gradcam(model, clone=True)

Generate heatmap with GradCAM
cam = gradcam(Y[test], [X1[test], X2[test]], penultimate_layer=-1)

ValueError: Score object must be callable! [mean_squared_error]

keisen commented 3 years ago

Hi @linbingru

The first argument of Gradcam#__call__() MUST be the score function as follows.


function score(outputs):
    # TODO calculate score values
    return score_values

cam = gradcam(score, (X1, X2), penultimate_layer=-1)

Please see the API document and examples for details.

Thanks!

linbingru commented 3 years ago

Hi,@keisen

Thank you very much for solving my question! I slove the above problem, however, the new problem comes out.

"The result calculated by saliency map it can represent the weight position. However, the result calculated by gradcam was decreasing by value."

from tf_keras_vis.gradcam import Gradcam
from tf_keras_vis.utils.model_modifiers import ReplaceToLinear

# score function
def score_function(output):
    return output[:, 0]

# attention layer
layer_idx='dense_3'

# cam object
#replace2linear = ReplaceToLinear()
gradcam = Gradcam(model, clone=True)

# Generate heatmap with GradCAM
cam = gradcam(score_function, [X2[test], X1[test]], penultimate_layer='dense_3')

And when I try different score_function the result goes zero.

print(saliency_map)
[array([[0.03965255, 0.06007694, 0.08179716, 0.08468853, 0.15592117,
         0.16889493, 0.21754322, 0.19366294, 0.2498665 , 0.26016768,
         0.36875371, 0.434514  , 0.48417194, 0.43154544, 0.54129211,
         0.50707816, 0.61562472, 0.68896445, 0.76950206, 0.82405041,
         0.84028072, 0.80198821, 0.76708105, 0.99999993, 0.76179289,
         0.8125057 , 0.84014564, 0.84142979, 0.86374242, 0.70754946,
         0.78490675, 0.85101713, 0.95127031, 0.80641341, 0.69927845,.......]],dtype=float32)]

print(cam)
[array([[1.00000000e+00, 9.53662038e-01, 9.07324016e-01, 8.60986114e-01,
         8.14648151e-01, 7.68310249e-01, 7.21972227e-01, 6.75634325e-01,
         6.29296362e-01, 5.82958400e-01, 5.36620438e-01, 4.90282506e-01,
         4.43944544e-01, 3.97606581e-01, 3.51268649e-01, 3.10990214e-01,
         3.01009357e-01, 2.91028500e-01, 2.81047612e-01, 2.71066755e-01,
         2.61085898e-01, 2.51105040e-01, 2.41124198e-01, 2.31143340e-01,
         2.21162468e-01, 2.11181596e-01, 2.01200753e-01, 1.91219881e-01,
         1.81239024e-01, 1.71258166e-01, 1.62492141e-01, 1.56155795e-01,
         1.49819434e-01, 1.43483087e-01, 1.37146741e-01, 1.30810395e-01,
         1.24474056e-01, 1.18137702e-01, 1.11801349e-01, 1.05465010e-01,
         9.91286561e-02, 9.27923024e-02, 8.64559487e-02, 8.01196098e-02,
         7.37832561e-02, 6.95410892e-02, 6.73930794e-02, 6.52450845e-02,
         6.30970821e-02, 6.09490909e-02, 5.88010848e-02, 5.66530824e-02,.......]],dtype=float32)]

Best wish, Thanks for helping me to solve my issue.

keisen commented 3 years ago

Hi, @linbingru .

Saliency is visualized by the gradients with respect to input values, Gradcam is visualized by the gradients with respect to the activation of the convolution layer you specified. (Because you specified dense_3 above, Gradcam refers to the convolutional layer just before it.) These will highlight similar positions basically, however, if the difference is big, the model might NOT be trained enough or there are some other problems.

Could you submit the result of model.summary() and the code snippet for reproducing the issue?

Thanks!

keisen / tf-keras-vis

multiple inputs CNN based regression problem #76