Occlusion sensitivity works for only one class, but shows error for other class in cat_vs_dogs dataset from tensorflow

[rao208]

I am trying to apply occlusion sensitivity on the "cats_vs_dog" dataset. First, the model is trained as given in the link https://www.tensorflow.org/tutorials/images/transfer_learning and then the trained model and the sample image is loaded to carry out occlusion sensitivity.

In an amateur attempt, the following lines were added in the code for tf-explain.

img, lbl = format_example(image, label) #img and lbl are tensor. Please see the above link for format_example funciton
img = img.numpy()
data = ([img], None)
print("Data", data)
classindex = lbl.numpy()
print("classindex", type(classindex))

# Instantiation of the explainer
explainer = OcclusionSensitivity()

# Call to explain() method
output = explainer.explain(data, new_model,  class_index = classindex, patch_size = 4)
# Save output
explainer.save(grid = output, output_dir = ".",
               output_name = "os-" + datetime.now().strftime("%Y%m%d-%H%M%S") + ".png")

It works fine for class 0 i.e. cats but fails to produce results for class 1 i.e. dogs.

Traceback (most recent call last):

File "C:\Users\Vanditha Rao.spyder-py3\os_tf_explain_cats_vs_dogs.py", line 107, in output = explainer.explain(data, new_model, class_index = classindex, patch_size = 4)

File "D:\Anaconda\envs\master_thesis\lib\site-packages\tf_explain\core\occlusion_sensitivity.py", line 49, in explain for image in images

File "D:\Anaconda\envs\master_thesis\lib\site-packages\tf_explain\core\occlusion_sensitivity.py", line 49, in for image in images

File "D:\Anaconda\envs\master_thesis\lib\site-packages\tf_explain\core\occlusion_sensitivity.py", line 102, in get_sensitivity_map prediction[class_index] for prediction in predictions

File "D:\Anaconda\envs\master_thesis\lib\site-packages\tf_explain\core\occlusion_sensitivity.py", line 102, in prediction[class_index] for prediction in predictions

IndexError: index 1 is out of bounds for axis 0 with size 1

The complete code is:

import matplotlib.pyplot as plt
import tensorflow as tf
import tensorflow_datasets as tfds
import urllib3
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
from tensorflow.keras.models import load_model
from tensorflow.keras.models import model_from_json
from tf_explain.core.occlusion_sensitivity import OcclusionSensitivity
from datetime import datetime

tfds.disable_progress_bar()

(raw_train, raw_validation, raw_test), metadata = tfds.load('cats_vs_dogs',
                                                            split=[
                                                                tfds.Split.TRAIN.subsplit(tfds.percent[:80]),
                                                                tfds.Split.TRAIN.subsplit(tfds.percent[80:90]),
                                                                tfds.Split.TRAIN.subsplit(tfds.percent[90:]),
                                                                ],
                                                            with_info=True,
                                                            shuffle_files=True,
                                                            as_supervised=True,)

get_label_name = metadata.features['label'].int2str
classes = metadata.features["label"].names

print("done")

for image, label in raw_train.take(1):
    print("Label: %d" % label.numpy())
    plt.figure()
    plt.imshow(image)
    plt.title(get_label_name(label))

IMG_SIZE = 160 # All images will be resized to 160x160

def format_example(image, label):
  image = tf.cast(image, tf.float32)
  image = (image/127.5) - 1
  image = tf.image.resize(image, (IMG_SIZE, IMG_SIZE))
  return image, label

# Apply this function to each item in the dataset using the map method:

train = raw_train.map(format_example)
validation = raw_validation.map(format_example)
test = raw_test.map(format_example)

BATCH_SIZE = 32
SHUFFLE_BUFFER_SIZE = 1000

train_batches = train.shuffle(SHUFFLE_BUFFER_SIZE).batch(BATCH_SIZE)
validation_batches = validation.batch(BATCH_SIZE)
test_batches = test.batch(BATCH_SIZE)

for image_batch, label_batch in train_batches.take(10):
    pass

print("image_batch.shape", image_batch.shape)
print("label_batch.shape", label_batch.shape)

# Loads the weights

# Model reconstruction from JSON file

with open('./model/model_cats_vs_dogs.json', 'r') as f:

    new_model = model_from_json(f.read())

new_model = load_model('./model/model_cats_vs_dogs.h5')
checkpoint_path = "./training_2_weights/cp.ckpt"
new_model.load_weights(checkpoint_path)

new_model.summary()

# Re-evaluate the model

test_steps=20
loss,acc = new_model.evaluate(test_batches, verbose=1, steps = test_steps)
print("Restored model, accuracy: {:5.2f}%".format(100*acc))

img, lbl = format_example(image, label)
print(img.shape, type(img))
img = img.numpy()

print(type(img))

data = ([img], None)
print("Data", data)
classindex = lbl.numpy()
print("classindex", type(classindex))

# Instantiation of the explainer
explainer = OcclusionSensitivity()

# Call to explain() method
output = explainer.explain(data, new_model,  class_index = classindex, patch_size = 4)
# Save output
explainer.save(grid = output, output_dir = ".",
               output_name = "os-" + datetime.now().strftime("%Y%m%d-%H%M%S") + ".png")

Also, the occlusion sensitivity method works fine on the pre-trained model like MobileNet or VGG but doesn't work on the own model.

[ShreyashSoni]

facing the same issue, any solution?

[rao208]

@ShreyashSoni Yes, the problem was I wasn't using the right activation function to get the probability score for both the classes. What I mean is, if you use a softmax activation function in the final layer (even if the dataset has two labels), you will get the probability score for both of them whereas if you use any other activation function then you will just get the probability for one class.

It is important to have the probability score for both the class because when we call tf explain Occlusion Sensitivity, we must specify the class index.

output = explainer.explain(data, new_model, class_index = classindex, patch_size = 4)

I hope it helps!

[ShreyashSoni]

@rao208 thanks for the explanation. this helps