Compute saliency maps for instance segmentation models

[AndreaPi]

I would like to understand why my model detected a certain object (mask) as having class A rather than class B. Is there a way to do it with MMDetection? If not, are there libraries that can process MMDetection output to compute a saliency map? Computing saliency maps for image classification tasks is easy, but for instance segmentation tasks things get more complicated.

[RangiLyu]

MMDetection has not supported analyzing saliency maps yet because different models have different structures. But we supported GradCAM and GradCAM++ and other saliency map visualization methods in MMYOLO for the YOLO series. Maybe you can refer to that and apply it to your model. https://github.com/open-mmlab/mmyolo/blob/main/demo/boxam_vis_demo.py

[AndreaPi]

Hi, thanks for your answer.

MMDetection has not supported analyzing saliency maps yet because different models have different structures.

It would be nice to have them at least for the more established methods, though. For example, Mask-RCNN...I'll create a feature request.

But we supported GradCAM and GradCAM++ and other saliency map visualization methods in MMYOLO for the YOLO series. Maybe you can refer to that and apply it to your model. https://github.com/open-mmlab/mmyolo/blob/main/demo/boxam_vis_demo.py

I had a quick look at the code. I have a few questions:

1. The docstring says that the code is still experimental, is that correct?
2. These saliency maps are for object detection only, rather than for instance segmentation, right?
3. I see dependencies from a few mmyolo modules:

from mmyolo.utils import register_all_modules
from mmyolo.utils.boxam_utils import (BoxAMDetectorVisualizer,
                                      BoxAMDetectorWrapper, DetAblationLayer,
                                      DetBoxScoreTarget, GradCAM,
                                      GradCAMPlusPlus, reshape_transform)
from mmyolo.utils.misc import get_file_list

I would have to reimplement them in MMDetection, right?

[AndreaPi]

@RangiLyu I see that @hhaAndroid wrote a PR to incorporate Grad-CAM for Mask-RCNN!

https://github.com/open-mmlab/mmdetection/pull/7987

That would be more than enough for me! Do you think there's a chance the PR will be merged? cc @ZwwWayne

[RangiLyu]

@hhaAndroid @ZwwWayne Maybe we can merge this feature into the projects folder?

[AndreaPi]

Any news on the possibility to merge https://github.com/open-mmlab/mmdetection/pull/7987?

[hhaAndroid]

@AndreaPi #7987 was developed based on earlier 2.x versions. There is a latest version in MMYOLO, if it is necessary, I or community user can propose a pr to migrate it to mmdet 3.x.

[AndreaPi]

@hhaAndroid thanks for the answer. My comments below:

There is a latest version in MMYOLO

I need Grad-CAM for Mask-RCNN. The one in MMYOLO is for YOLO.

if it is necessary, I or community user can propose a pr to migrate it to mmdet 3.x.

I think it would be better to simply migrate it to the latest version of mmdet 2.x because I'm pretty sure it's more stable than https://github.com/open-mmlab/mmdetection/releases/tag/v3.0.0rc5. But if you really want to migrate it to 3.x, that will be still better than no PR :-)

[AndreaPi]

@hhaAndroid MMDetection 3.0.0 is officially out. What about migrating #7987 to it?

[AndreaPi]

Any news about the possibility to migrate the latest version of saliency maps in MMYOLO, to MMDetection 3.0? Should I open a new issue and close this one?

@AndreaPi #7987 was developed based on earlier 2.x versions. There is a latest version in MMYOLO, if it is necessary, I or community user can propose a pr to migrate it to mmdet 3.x.