Error occurred in GradCAM++ when the data type of model-outputs and gradients are different.

[keisen]

Here's a minimal example that is giving the above error:

import numpy as np
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import applications, layers
from tensorflow.keras.mixed_precision import experimental as mixed_precision
from tf_keras_vis.gradcam import GradcamPlusPlus

policy = mixed_precision.Policy("mixed_float16")
mixed_precision.set_policy(policy)

base_model = applications.MobileNet(
    input_shape=[224, 244, 3],
    include_top=False,
    weights=None,
)
layer = base_model.output
layer = layers.Flatten(name="flatten")(layer)
layer = layers.Dense(2, dtype=tf.float32)(layer)
model = keras.models.Model(inputs=base_model.input, outputs=layer)
data = np.empty(model.input.shape[1:])

def model_modifier(model):
    model.layers[-1].activation = keras.activations.linear
    return model

def loss(output):
    return [o[0] for o in output]

GradcamPlusPlus(model, model_modifier)(loss, data)
print("Done.")

Originally posted by @bersbersbers in https://github.com/keisen/tf-keras-vis/issues/41#issuecomment-745512310

[bersbersbers]

54c4def11b6cd83153fc6734fd5ae1cb2dde7802 is working great, thank you!

[keisen]

Hi, @bersbersbers .

Although I've tried to support mixed-precision in tf-keras-vis v0.6.0 regardless tensorflow versions, unfortunately, I gave up to fix this issue that relates experimentally mixed-precision (tensorflow < 2.4.0), due to my time. I don't have a time to realize consistent specification about supporting mixed-precision of all versions in tf-keras-vis. So tf-keras-vis v0.6.0 will only support mixed-precision of tensorflow 2.4.0 or higher.

Coming soon, I'm going to close this issue, but feel free please re-open when you need.

Thanks!

[bersbersbers]

Hi @keisen, I am using TF 2.4.1 anyway and will transition to 2.5.0 as soon as it is released - so no worries, I'll be a happy man when these issues are fixed for TF>=2.4.0 :)

[bersbersbers]

This is not working in 66132db3:

# pip install tensorflow==2.4.1 git+https://github.com/keisen/tf-keras-vis@66132db3
import tensorflow as tf
from tf_keras_vis import gradcam

tf.keras.mixed_precision.set_global_policy("mixed_float16")
base_model = tf.keras.applications.MobileNet(
    include_top=False,
    input_shape=(32, 32, 3),
    weights=None,
)
layer = base_model.output
layer = tf.keras.layers.Flatten(name="flatten")(layer)
layer = tf.keras.layers.Dense(2, dtype=tf.float32)(layer)
model = tf.keras.models.Model(inputs=base_model.input, outputs=layer)
data = tf.zeros(model.input.shape[1:])
loss = lambda output: sum(output)
gradcam.GradcamPlusPlus(model)(loss, data)
print("Done.")

Input to reshape is a tensor with 2 values, but the requested shape requires a multiple of 1024 [Op:Reshape]

This problem is independent of the mixed policy, but I assume that once that is fixed, the same thing will happen as in #41.

[keisen]

Hi @bersbersbers . Thank you so much for pointing it out. I've fixed it (See for details: 5d3185bbb4a0fd69ecd633d692a12e0489bfec73).

Thanks!