Exception: This is not supposed to happen! ERROR with ResNet50 transfer learning

[saberAL]

Hello, I was trying to use innvestigate in combination with a finetuned Resnet50 model which I created in keras, by taking off the last layer and adding a dense layer with sigmoid activation function. I loaded the model and created the analyzer according to tutorial, the problem is when I try to analyze a given image, I run into an Exception that only states that it is not supposed to happen. Maybe I am using the analyzer in a wrong way or maybe it doesn't work with my model. This my model:

resnet = ResNet50(
        include_top = False,
        weights = 'imagenet',
        pooling = "avg",
)

#freeze the  layers
for layer in resnet.layers:
        layer.trainable = True

output_layer = layers.Dense(1, activation = "sigmoid")

model = keras.models.Sequential([
        resnet, output_layer
    ])

and this is the analayzer:

model = keras.models.load_model("resnet50_model.h5")
model.summary()

image_height = image_width = 224
batch_size = 32

#Load test data
test_datagen = ImageDataGenerator(rescale = 1.0/255)
test_gen = test_datagen.flow_from_directory(
        "data/test/",
        target_size = (image_height, image_width),
        batch_size = batch_size,
        color_mode = "rgb",
        class_mode = "binary",
        shuffle = True,
        seed=1
)

#Select the first test image 
image = x[1]

plt.imshow(image, interpolation = "nearest")
plt.show()

#Take off the output layer
model_wo_ol = iutils.keras.graph.model_wo_softmax(model)

#Create gradient analyzer
gradient_analyzer = innvestigate.create_analyzer("gradient", model_wo_ol)

#Apply gradient analyzer
gradient_analysis = gradient_analyzer.analyze(image)

# Aggregate along color channels and normalize to [-1, 1]
gradient_analysis = gradient_analysis.sum(axis=np.argmax(np.asarray(gradient_analysis.shape) == 3))
gradient_analysis /= np.max(np.abs(gradient_analysis))
# Plot and save image
plt.imshow(gradient_analysis[0], cmap="seismic", clim=(-1, 1))
plt.axis('off')
plt.show()
plt.savefig("gadient.png")

that gives the following tracetrack:

Thank you.

[rachtibat]

Hi,

at least for LRP Methods, ResNet is not supported in innvestigate 1.0. Version 2.0 is still under development. I recommend trying out alternative XAI solutions.

Bets

[saberAL]

Hi, could you please tell me which ones are supported? Thank you

[rachtibat]

Hi, normal Conv Networks like VGG16. Best

[saberAL]

Thank you for answering me, but I tried VGG16 and loaded my finetuned model:

vgg = VGG16(input_shape = (image_height, image_width, 3), weights = 'imagenet',include_top = False)

for layer in vgg.layers:
        layer.trainable=True  

x = Flatten()(vgg.output)
x = Dense(2048, activation='relu')(x)
x = Dropout(0.5)(x)
prediction = Dense(1, activation='sigmoid')(x)

model = Model(input = vgg.input, output = prediction)

where image_heigth=image_width=224, but I get valueError when checking input: Here the traceback:

[rachtibat]

Hi,

did you feed in a batch [Batch_size, 224, 224, 3] ? You need to add the batch dimension.

You should first try to define a simple LeNet from Scratch. If it works, but VGG16 not, then you can try out to "unpack" the VGG model i.e. something like this

for layer in model.layers: x = layer(x)

Because if you do model.layers on a model that is comprised of submodels, you do not get the actual layers, but the keras.model instances. Innvestigate might be confused.

Unfortunately, I can not help you furthermore on this as I am not familiar with the inner workings of innvestigate 1.0.

Best

[saberAL]

Yes but it gives me error because input_shape must be a triple tuple. Thank you so much for the help, I will try it.

[rachtibat]

No worries, And do not forget: innvestigate.analyze( images, ...) the images have to be [batch, width, height, channel] dimension

[saberAL]

I solved the problem, I was selecting the first image of the batch. Thank you so much.

[rachtibat]

nice!