What is different between the models?

[xBorja042]

Hello, I have a suggestion for you. Since you have implement the methods so you understand very well the ideas behind these methods I would recommend you to explain what are the main difference between the methods on the examples notebook. Just a short detail, that would very great.

Thanks and kind regards,

Borja.

[keisen]

@xBorja042 , Thank you for your great idea! Currently I'm writing a web document for tf-keras-vis, so I'm going to tackle to do that. However it will take a time, If you have any question, please feel free to ask us in this issue.

Thanks!

[xBorja042]

@keisen Thanks a lot for your response. I need to read deeply the paper for ScoreCam but I do not really get the point between the differences of gradcam and gradcam++. Also, I would like to understand the purpose of the utility of the score list.

Thanks a lot and kind regards,

Borja.

[keisen]

I need to read deeply the paper for ScoreCam but I do not really get the point between the differences of gradcam and gradcam++.

The biggest difference between ScoreCAM and GradCAMs is whether it needs the gradient calculation.

Also, I would like to understand the purpose of the utility of the score list.

the score list means tf_keras_vis.utils.Score classes? If so, because the document is now writting, please refer to the example notebooks below.

https://github.com/keisen/tf-keras-vis/tree/master/examples

Thanks!

[xBorja042]

Thanks @keisen

I understand what score is. What I do not understand is that why the methods need it as input and what to do if I am not using VGG16. Thanks a lot

[keisen]

@xBorja042 , could you clarify what the problem or the situation are? Or could you submit the code snippet to reproduce the problem? If you could, it would greatly help our discussion!

[xBorja042]

I have not experienced the problem yet. But I want to use your library to explain the experts how the model is working @keisen.

I will try to explain to you better:

Let us suppose that I have written my own model based on CONVs layers. The problem is that I do not have this part of the code:

score = CategoricalScore([1, 294, 413])

This is because I am not going to use VGG16 but my own model to classify. The thing is if it is possible to adapt your library for ad-hoc models.

Thanks for your patience and kind responses.

[keisen]

@xBorja042 , the code below is a part of the example in README.md.

# Create GradCAM++ object
gradcam = GradcamPlusPlus(YOUR_MODEL_INSTANCE,
                          model_modifier=ReplaceToLinear(),
                          clone=True)

# Generate cam with GradCAM++
cam = gradcam(CategoricalScore(CATEGORICAL_INDEX),
              SEED_INPUT)

Is the question that, also when YOUR_MODEL_INSTANCE above is your model, whether you can use the CategoricalScore class? If so, the answer is YES. If you want to implement the custom CategoricalScore class, please let me know your model architecture. If neither, I'm sorry to bother you, please tell me again another way.

Thanks!

[xBorja042]

@keisen Do not say sorry! I feel very gratefull for your help. I am not sure that I have explained well.

My question is about what to do when the model does not have a score like this:

score = CategoricalScore([1, 294, 413])

The input for CategoricalScore is just the correct values assigned for the images? If so, I need to know what images am I using. Is it not possible to use the methods you have implemented without the score variable?

Thanks again!

[keisen]

@xBorja042 , okay, let's go next!

The input for CategoricalScore is just the correct values assigned for the images?

[1, 294, 413] are indexes corresponding to model output. That's, they are indexes corresponding to somethings you want to visualize.

Here, please see the code that is a part of Attention Example Notebok. The implement below is equivalent to CategoricalScore([1, 294, 413]).

def score_function(output):
    # The `output` variable refers to the output of the model,
    # so, in this case, `output` shape is `(3, 1000)` i.e., (samples, classes).
    return (output[0][1], output[1][294], output[2][413])

As you see above, it just returns the values of model output corresponding to [1, 294, 413]. When you visualize with your model, you have to specify the index of your model output with CategoricalScore or your own function such above.

Is it not possible to use the methods you have implemented without the score variable?

So, all visualization algorisms NEED the score. If we don't specify any score, it's that there is nothing we want to visualize.

Thanks!

[xBorja042]

Good morning @keisen . Thanks a lot for your help!

I think I do understand the idea. So, let us suppose that that I have a Binary classificacion model and the model outputs for 2 samples are [0.1, 0.9] and [0.9, 0.1] and the true category results are 0 and 1. In that case my score variable is just:

 score = BinaryScore([model.predict(samples)[0],model.predict(samples)[1]])

[keisen]

the model outputs for 2 samples are [0.1, 0.9] and [0.9, 0.1] and the true category results are 0 and 1.

In binary classification, when the model output shape is (batch_size, 2), please use CategoricalScore class to specify index of the category you want to visualize. Please use BinaryScore class only when the model output shape is (batch_size, 1), that's, the sigmoid function is applied to the model output.

So, let us suppose that that I have a Binary classificacion model and the model outputs for 2 samples are [0.1, 0.9] and [0.9, 0.1] and the true category results are 0 and 1. In that case my score variable is just:

So, in this case, if you want to visualize the CAMs or Saliency maps corresponding to the ture category results, you can implement the score function as follows:

score = CategoricalScore([0, 1])

Or,

function score_function(output):
    return (output[0][0], output[1][1])

Both implements above are the same! Then, you can use them as follows:

gradcam = Gradcam(model, ...)
cam_image = gradcam(score, SEED_INPUT, ...)
# or, cam_image = gradcam(score_function, SEED_INPUT, ...)

To understand how the score or tf-keras-vis work, I strongly recommend you to read the example notebooks of tf-keras-vis in examples folder and to experiment while modifying their code. (of course, before that, reading the papers of Gradcam, Gradcam++ and Scorecam would be also good!)

When you face any error that you can't solve, please come back here and feel free to ask us!

Thanks!

[xBorja042]

All right @keisen .

Thanks a lot for your help and sorry for the incoveniences. Have a good day!

[keisen]

@xBorja042 , no problem! I'm glad if you had fun with tf-keras-vis.

Thanks!