Problems implementing with my own model

[JeffreyShran]

I'm looking for some help please.

I have my own model for which I'd like to implement the saliency techniques here.... and thanks for supplying the code examples and documentation, it's really useful. Recreating your core example works flawlessly in Google Colab, however I now want to use my own model and images.

Firstly here is my code, notable things are that I dropped the preprocessimage function and used the same code I did to create the model and in call_model_function() I changed model(images) to model.predict(images) because the code failed there and I assumed my implementation should mirror the prediction part of the code I got working, but I'm unsure if that's correct? and finally I had to add im = im.reshape(256,256,1) right before moving into the gradient_saliency = saliency.GradientSaliency() stuff because it complained that my image was the wrong shape, which makes no sense to me as model.predict already worked prior to that?

Code:

model = tf.keras.models.load_model('/content/drive/MyDrive/model.h5')

class_idx_str = 'class_idx_str'
def call_model_function(images, call_model_args=None, expected_keys=None):
    target_class_idx =  call_model_args[class_idx_str]
    images = tf.convert_to_tensor(images)
    with tf.GradientTape() as tape:
        if expected_keys==[saliency.base.INPUT_OUTPUT_GRADIENTS]:
            tape.watch(images)
            _, output_layer = **model.predict(images) # I changed this to predict**
            output_layer = output_layer[:,target_class_idx]
            gradients = np.array(tape.gradient(output_layer, images))
            return {saliency.base.INPUT_OUTPUT_GRADIENTS: gradients}
        else:
            conv_layer, output_layer = model(images)
            gradients = np.array(tape.gradient(output_layer, conv_layer))
            return {saliency.base.CONVOLUTION_LAYER_VALUES: conv_layer,
                    saliency.base.CONVOLUTION_OUTPUT_GRADIENTS: gradients}

file_path = ('/content/drive/MyDrive/dataset/01215.jpg')
im = cv.cvtColor(cv.imread(file_path), cv.COLOR_BGR2GRAY)
im = np.asarray(im, dtype=np.float32)
im = cv.resize(im, (img_height, img_width))
im = np.expand_dims(im, axis=0)
print(f'im expand: {im.shape}')

predictions = model.predict(im, batch_size=1, verbose='silent')
prediction_class = np.argmax(predictions[0])
call_model_args = {class_idx_str: prediction_class}
print(f'call_model_args: {call_model_args}')

class_names = ['active','inactive']
prediction = class_names[np.argmax(predictions)]
percentage = int(np.max(predictions, axis=-1)*100)
print(f'{file_path} - {prediction} - {percentage}% - Prediction class: {str(prediction_class)}')

im = im.reshape(256,256,1) # I did this because the subsequent code complained about the shape. (Which makes no sense to me as model.predict already worked above?)

Output:

im expand: (1, 256, 256) call_model_args: {'class_idx_str': 1} /content/drive/MyDrive/dataset/01215.jpg - inactive - 98% - Prediction class: 1

So this is great and works as expected so far. Now I try to run the next part of the core example code and I get an error that I do not understand.

# Construct the saliency object. This alone doesn't do anything.
gradient_saliency = saliency.GradientSaliency()

# Compute the vanilla mask and the smoothed mask.
vanilla_mask_3d = gradient_saliency.GetMask(im, call_model_function, call_model_args)
smoothgrad_mask_3d = gradient_saliency.GetSmoothedMask(im, call_model_function, call_model_args)

# Call the visualization methods to convert the 3D tensors to 2D grayscale.
vanilla_mask_grayscale = saliency.VisualizeImageGrayscale(vanilla_mask_3d)
smoothgrad_mask_grayscale = saliency.VisualizeImageGrayscale(smoothgrad_mask_3d)

# Set up matplot lib figures.
ROWS = 1
COLS = 2
UPSCALE_FACTOR = 10
P.figure(figsize=(ROWS * UPSCALE_FACTOR, COLS * UPSCALE_FACTOR))

# Render the saliency masks.
ShowGrayscaleImage(vanilla_mask_grayscale, title='Vanilla Gradient', ax=P.subplot(ROWS, COLS, 1))
ShowGrayscaleImage(smoothgrad_mask_grayscale, title='SmoothGrad', ax=P.subplot(ROWS, COLS, 2))

---

LookupError Traceback (most recent call last) in 3 4 # Compute the vanilla mask and the smoothed mask. ----> 5 vanilla_mask_3d = gradient_saliency.GetMask(im, call_model_function, call_model_args) 6 smoothgrad_mask_3d = gradient_saliency.GetSmoothedMask(im, call_model_function, call_model_args) 7

3 frames /usr/local/lib/python3.8/dist-packages/tensorflow/python/ops/gradients_util.py in _GradientsHelper(ys, xs, grad_ys, name, colocate_gradients_with_ops, gate_gradients, aggregation_method, stop_gradients, unconnected_gradients, src_graph) 637 grad_fn = func_call.python_grad_func 638 else: --> 639 raise LookupError( 640 "No gradient defined for operation" 641 f"'{op.name}' (op type: {op.type}). "

LookupError: No gradient defined for operation'IteratorGetNext' (op type: IteratorGetNext). In general every operation must have an associated @tf.RegisterGradient for correct autodiff, which this op is lacking. If you want to pretend this operation is a constant in your program, you may insert tf.stop_gradient. This can be useful to silence the error in cases where you know gradients are not needed, e.g. the forward pass of tf.custom_gradient. Please see more details in https://www.tensorflow.org/api_docs/python/tf/custom_gradient.

I'm not great a writing code so I've clearly done something stupid, can anyone assist me in what the problem might be please? I'm also open to pointers on any other aspect too. If it helps this is how I created the model:

model = tf.keras.Sequential([
  tf.keras.layers.Rescaling(1./255),
  tf.keras.layers.Conv2D(32, 3, activation='relu'),
  tf.keras.layers.MaxPooling2D(),
  tf.keras.layers.Conv2D(32, 3, activation='relu'),
  tf.keras.layers.MaxPooling2D(),
  tf.keras.layers.Conv2D(32, 3, activation='relu'),
  tf.keras.layers.MaxPooling2D(),
  tf.keras.layers.Flatten(),
  tf.keras.layers.Dense(128, activation='relu'),
  tf.keras.layers.Dense(2,activation='softmax') #activation change
])