Add third saliency map interpretation notebook

View / edit / reply to this conversation on ReviewNB

negvet commented on 2024-07-29T16:26:08Z ----------------------------------------------------------------

This notebook shows which saliency map are generated in different scenarios, in case of correct and confident prediction, in case of low-confident prediction and in case of mixed up classes.

What about below:

This notebook shows how to use saliency maps to evaluate and debug model reasoning.

For example, it might be important to make sure that the model is using relevant features (pixels) to make a correct prediction (e.g. it might be desirable that the model is not relying on X class features to predict Y class). On the other side, it is valuable to observe which features are used when the model is wrong.

Below, we present examples of saliency map analysis for the following cases: correct and highly-confident prediction, correct and low-confident prediction, wrong prediction.

review-notebook-app[bot] commented 1 month ago

View / edit / reply to this conversation on ReviewNB

negvet commented on 2024-07-30T11:48:26Z ----------------------------------------------------------------

Line #1.    # Download 330 MB of 320 px ImageNet subset with dogs breeds

dog breeds

review-notebook-app[bot] commented 1 month ago

View / edit / reply to this conversation on ReviewNB

negvet commented on 2024-07-30T11:48:27Z ----------------------------------------------------------------

Line #2.    img_data_formats = (".jpg", ".jpeg", ".JPEG", ".gif", ".bmp", ".tif", ".tiff", ".png")

Do we need all of those formats? if possible, let's simplify it to make notebook more concise.

review-notebook-app[bot] commented 1 month ago

View / edit / reply to this conversation on ReviewNB

negvet commented on 2024-07-30T11:48:27Z ----------------------------------------------------------------

The model is from timm, right? then, let's be a bit more specific about its origin:

In this notebook, for demonstration purposes, we'll use an already converted to IR model "mobilenetv3_large_100.ra_in1k", from timm (PyTorch Image Models).

review-notebook-app[bot] commented 1 month ago

View / edit / reply to this conversation on ReviewNB

negvet commented on 2024-07-30T11:48:28Z ----------------------------------------------------------------

Line #2.        # Implementing own pre-process function based on model's implementation

let's make it a docstring

review-notebook-app[bot] commented 1 month ago

View / edit / reply to this conversation on ReviewNB

negvet commented on 2024-07-30T11:48:29Z ----------------------------------------------------------------

Line #5.        # Normalization and scaling for timm model

Normalization, specific for timm model

review-notebook-app[bot] commented 1 month ago

View / edit / reply to this conversation on ReviewNB

negvet commented on 2024-07-30T11:48:29Z ----------------------------------------------------------------

Line #19.        # Process model prediction

also a docstring

review-notebook-app[bot] commented 1 month ago

View / edit / reply to this conversation on ReviewNB

negvet commented on 2024-07-30T11:48:30Z ----------------------------------------------------------------

Line #20.        prediction_processed = softmax(x)

return softmax(x)

should it be "return softmax(x)[0]" to remove the batch dimension?

review-notebook-app[bot] commented 1 month ago

View / edit / reply to this conversation on ReviewNB

negvet commented on 2024-07-30T11:48:31Z ----------------------------------------------------------------

let's clean this cell output

GalyaZalesskaya commented on 2024-07-30T13:23:13Z ----------------------------------------------------------------

This is the logger.info output that for some reason can't be captured by usual "%%capture" magic function. The other way to clean output is to add setLevel for logger, but it will require additional logger import, making the overall decision not concise. Maybe we can leave it as it is.

review-notebook-app[bot] commented 1 month ago

View / edit / reply to this conversation on ReviewNB

negvet commented on 2024-07-30T11:48:32Z ----------------------------------------------------------------

from this it might seems that we extract some info from model meta information. this is not correct.

If label_names is not provided to the explainer call, the saved saliency map will have the predicted class index, not the name. For example, image_name_target_167.jpg instead of image_name_target_English_foxhound.jpg.

To conveniently view label names in saliency maps, we provide ImageNet label names information to the explanation call.

GalyaZalesskaya commented on 2024-07-30T13:00:46Z ----------------------------------------------------------------

Thank you for noticing. It wasn't updated after we changed the implementation in XAI. Fixed the same issue in 2nd deep-dive notebook.

review-notebook-app[bot] commented 1 month ago

View / edit / reply to this conversation on ReviewNB

negvet commented on 2024-07-30T11:48:32Z ----------------------------------------------------------------

Please clean cell output

review-notebook-app[bot] commented 1 month ago

View / edit / reply to this conversation on ReviewNB

negvet commented on 2024-07-30T11:48:33Z ----------------------------------------------------------------

Line #1.    # Get ImageNet label names to add them to explanations

review-notebook-app[bot] commented 1 month ago

View / edit / reply to this conversation on ReviewNB

negvet commented on 2024-07-30T11:48:34Z ----------------------------------------------------------------

Line #11.            ],  # Also label indexes [206, 258] are possible as target

indices

review-notebook-app[bot] commented 1 month ago

View / edit / reply to this conversation on ReviewNB

negvet commented on 2024-07-30T11:48:34Z ----------------------------------------------------------------

Line #1.    def get_model_predictions(conf_thr=0.1):

let's add a type hints and docstring

review-notebook-app[bot] commented 1 month ago

View / edit / reply to this conversation on ReviewNB

negvet commented on 2024-07-30T11:48:35Z ----------------------------------------------------------------

Line #28.            # to return predicted scores for saliency maps as well

Let's keep it as it for now, but let's update explanation.save to support it

_GalyaZalesskaya commented on 2024-07-30T13:37:19Z_ ----------------------------------------------------------------

We'll plan it for OVXAI 1.1 :)

review-notebook-app[bot] commented 1 month ago

View / edit / reply to this conversation on ReviewNB

negvet commented on 2024-07-30T11:48:36Z ----------------------------------------------------------------

Line #1.    # Function to show result images for each use case

Function to show result saliency maps for each use case

review-notebook-app[bot] commented 1 month ago

View / edit / reply to this conversation on ReviewNB

negvet commented on 2024-07-30T11:48:37Z ----------------------------------------------------------------

this might be moved above for clarity.

also what do you think of expanding the names, e.g. tp_two_preds -> tp_two_predictions or true_positive_two_predictions

negvet commented 1 month ago

todo @negvet: review results

GalyaZalesskaya commented 1 month ago

Thank you for noticing. It wasn't updated after we changed the implementation in XAI. Fixed the same issue in 2nd deep-dive notebook.

View entire conversation on ReviewNB

GalyaZalesskaya commented 1 month ago

View entire conversation on ReviewNB

GalyaZalesskaya commented 1 month ago

Will plan it for OVXAI 1.1 :)

View entire conversation on ReviewNB

review-notebook-app[bot] commented 1 month ago

View / edit / reply to this conversation on ReviewNB

negvet commented on 2024-07-30T14:42:24Z ----------------------------------------------------------------

In these cases of True positive high confidence, the saliency maps are considered to be canonical.

There is not canonical or high quality saliency maps. Even with True positive high confidence the model might use wrong features to predict correct class.

I do understand that we need to be very careful with those interpretation. Let me try:

What about:

In the case of True positive high confidence the model predicts correct class and is confident about its prediction.

Saliency map highlight features, that strongly contribute to the correct class. Meaning that those features are very silent for current class. We want to roughly estimate that the features are the right ones. From the above images, we see that dog face, nose, ears, and the general shape of the dog body usually contain the strongest features that the model correctly learn (which makes sense).

Also, another sign that the model learn the right features is that the classes are well distinguished by the model. Cat features are not used at all to predict Samoyed in the image 17737, which is the desired behavior.

_______

@Galya: ideally we would want to have some image here, where the model is confidently correct, but using wrong features. Such a model will not generalize well. But I understand that such a case is hard to find.

review-notebook-app[bot] commented 1 month ago

View / edit / reply to this conversation on ReviewNB

negvet commented on 2024-07-30T14:42:24Z ----------------------------------------------------------------

True positive low confidence basically means that required features are not well available or transformed. From saliency maps we see that model paying attention to the whole object, trying to make a decision mostly based on high-level features.

review-notebook-app[bot] commented 1 month ago

View / edit / reply to this conversation on ReviewNB

negvet commented on 2024-07-30T14:58:32Z ----------------------------------------------------------------

We can see that it's not the problem of the model but rather the characteristic of the picture itself.

In the multiclass classification settings that we have here (with softmax applied), this can happen if features of a wrong class dominates the features of a wrong class. Also, this might indicate labeling error.

review-notebook-app[bot] commented 1 month ago

View / edit / reply to this conversation on ReviewNB

negvet commented on 2024-07-30T14:58:33Z ----------------------------------------------------------------

please correct images 634 <-> 374

GalyaZalesskaya commented 1 month ago

@eaidova Hi Ekaterina, you've been a great help during the preparation of notebooks for XAI. Can you please review?

review-notebook-app[bot] commented 1 month ago

View / edit / reply to this conversation on ReviewNB

eaidova commented on 2024-08-01T05:07:09Z ----------------------------------------------------------------

is it possible to provide link on dataset info?

review-notebook-app[bot] commented 1 month ago

View / edit / reply to this conversation on ReviewNB

eaidova commented on 2024-08-01T05:07:09Z ----------------------------------------------------------------

Line #2.    model = ov.Core().read_model(model_xml_path)

you already have core instance above (where device selected), you can use it for model reading

review-notebook-app[bot] commented 1 month ago

View / edit / reply to this conversation on ReviewNB

eaidova commented on 2024-08-01T05:07:10Z ----------------------------------------------------------------

Line #7.

the last print is useful for debugging, but I think it does not provide a lot of info to less experienced users (may raise questions about why only 10, how fish names connected with my task if in notebook we use dataset to classify dog breeds e.t.c). Maybe it is better to add some info about that (in any format: as comment, print some title, in description before cell) to highlight that it is just examples for checking expected data format. Also maybe it will be good to check that below used labels are in provided list (flat-coated_retriever, Samoyed") and find their ids?

GalyaZalesskaya commented 1 month ago

@eaidova Thank you for your great and thoughtful comments, I applied them

openvinotoolkit / openvino_notebooks

Add third saliency map interpretation notebook #2240