How to select an specific output from a Multi Task Learning Model?

[SergioLeonardoMendes]

Hello Keisen,

First I'd like to thank you for sharing this project. I've already used the former keras-vis and I found your project exactly when I was developing a new code for TF 2.1.

I built a multi task model with 3 output layers and I'm not sure how to select each of this outputs before running the saliency and normalize methods.

Here is the outputs of my model (obtained via model.outputs command). The first and third outputs use binary_crossentropy loss while the second uses MSE loss.

[<tf.Tensor 'sex_3/Identity:0' shape=(None, 1) dtype=float32>,
 <tf.Tensor 'age_3/Identity:0' shape=(None, 1) dtype=float32>,
 <tf.Tensor 'autism_3/Identity:0' shape=(None, 1) dtype=float32>]

I had success running the code bellow (from your examples). But I failed when I needed to select a different output and loss than the last one.

def model_modifier(m):
    m.layers[-1].activation = tf.keras.activations.linear
saliency = Saliency(model, model_modifier)
loss = lambda output: K.mean(output[:, 0])

Thanks in advance!

[keisen]

Thank you for your question, and i'm sorry for the inconvenience that I have not published the API documents.

When the model has multiple outputs, you can use a different loss on each output by passing a list of losses. Could you please refer to below.

loss_1 = lambda output: ...
loss_2 = lambda output: ...
loss_3 = lambda output: ...
losses = [ loss_1, loss_2, loss_3 ]

saliency_map = saliency(losses, ...)

[SergioLeonardoMendes]

Hello Keisen,

Thank you for your quick reply! The point is even if I pass a list of losses (like you did in the example) the output I get contains only one array (image) in the saliency_map. In this case, shouldn't I get more than one saliency_map (one for each output) ?

Sorry if I'm asking a dumb question. I'm trying to understand your code but as my knowledge is limited and I'm not having any lucky.

[keisen]

Saliency generates an attention map that appears how output value changes with respect to a small change in an input image pixels. So saliency map is always generated only one for each input of the model.

When you want to generate it for each output, you have to run Saliency the same times as the number of outputs. (Note, the losses without corresponding the output that you want to generate saliency map have to return zero.)

I wish you all the best.

[SergioLeonardoMendes]

When you want to generate it for each output, you have to run Saliency the same times as the number of outputs. (Note, the losses without corresponding the output that you want to generate saliency map have to return zero.)

Ok, but how do I specify the output of the model from which the salience will be generated? In my model I have different dense layers right before each output. Something like this:

INPUT -> COMMON_CONVOLUTIONAL_BODY -> OUTPUT_DENSE_BLOCK_1 (with 2 layers) -> OUTPUT1 OUTPUT_DENSE_BLOCK_2 (with 3 layers) -> OUTPUT2 OUTPUT_DENSE_BLOCK_3 (with 2 layers) -> OUTPUT3

Sorry for using this space to ask questions. I know it is probably not an issue of tf-keras-vis.

Thank you again!

[keisen]

I understood that your model is such as follow:

import tensorflow as tf
import tensorflow.keras.backend as K

def dense_block(x, num_of_layer=1):
    for i in range(num_of_layer):
        x = tf.keras.layers.Dense(5, activation='relu')(x)
    return x

inputs = tf.keras.layers.Input((12, 12, 3))
x = tf.keras.layers.Conv2D(1, 3, activation='relu')(inputs)
x1 = dense_block(x)
x2 = dense_block(x, num_of_layer=2)
x3 = dense_block(x)
output_1 = tf.keras.layers.Dense(1, name='sex', activation='sigmoid')(x1)
output_2 = tf.keras.layers.Dense(1, name='age')(x2)
output_3 = tf.keras.layers.Dense(1, name='autism', activation='sigmoid')(x3)
model = tf.keras.Model(inputs, [output_1, output_2, output_3])
model.summary()

If I understand it correctly, I believe, you can generate saliency maps for each the output of model as follow:

# By multiplying by zero, its gradients is to be zero.
zero_loss = lambda output: K.mean(output * 0.0)
# This function should be defined to satisfy something you want.
active_loss = lambda output: K.mean(output[:])

saliency = Saliency(model)
seed_input = ...
saliency_map_1 = saliency([active_loss, zero_loss, zero_loss], seed_input)
saliency_map_2 = saliency([zero_loss, active_loss, zero_loss], seed_input)
saliency_map_3 = saliency([zero_loss, zero_loss, active_loss], seed_input)

But if I misunderstand your model, could you point out it.

[SergioLeonardoMendes]

Great Keisen! You took exactelly the point. I was missing the trick of multiplying the "inactive losses" by zero. I'm sure your work is helping many researchers around the world.

I'm closing this topic and hope it can help someone with a similar question in future.

Thank you soo much!

[keisen]

I'm glad to hear that! If you would like, please click the Github Stars of this repository.

[SergioLeonardoMendes]

I'm glad to hear that! If you would like, please click the Github Stars of this repository.

For sure! I've already done it! Thanks!