search

jacobgil / pytorch-grad-cam

Advanced AI Explainability for computer vision. Support for CNNs, Vision Transformers, Classification, Object detection, Segmentation, Image similarity and more.
https://jacobgil.github.io/pytorch-gradcam-book
MIT License
10.68k stars 1.56k forks source link

grad-cam for regression #293

Closed TimSchim closed 2 years ago

TimSchim commented 2 years ago

Hi, I'm using ResNet18 for regression but couldn't find any information if grad-cam can be used for regression models too. Is this possible in general and if so, what would I pass to the 'targets' argument? Thanks a lot for any suggestions or hints, Tim

jacobgil commented 2 years ago

It's definitely possible. I made this post 6 years ago about regression: https://jacobgil.github.io/deeplearning/vehicle-steering-angle-visualizations

You need to define a custom target. This is an example of a target for classification: https://github.com/jacobgil/pytorch-grad-cam/blob/master/pytorch_grad_cam/utils/model_targets.py#L5

But instead of taking the output of a specific category, you could for example just take the output if you're interested in what makes the output higher, or multiply it by -1 if you're interested in what makes the output lower.

jacobgil commented 2 years ago

This new target can now be used for regression as well: https://github.com/jacobgil/pytorch-grad-cam/blob/master/pytorch_grad_cam/utils/model_targets.py#L21

Pass category=1 if you want to explain when the regression outputs go up, and 0 otherwise.

tjoliveira commented 2 years ago

@jacobgil Thank you very much for this amazing repo!

I'm also working on a regression problem and I tried following your recommendation to use ClassifierOutputSoftmaxTarget to define the target. However, since the regression model provides a single output, this line https://github.com/jacobgil/pytorch-grad-cam/blob/master/pytorch_grad_cam/utils/model_targets.py#L21:~:text=return%20torch.softmax(model_output%2C%20dim%3D%2D1)%5Bself.category%5D

throws the following error

IndexError: index 1 is out of bounds for dimension 0 with size 1

I've also tried to use RawScoresOutputTarget and it works. I'm a little confused about the interpretation of the heatmaps though. I think what you mentioned in your post about angle regression applies, i.e., if the gradient is large, it contributes to a large output and if it is lower, it contributes to low outputs. In my case, I'm regressing a value between 0-180 and the model is responding reasonably well in terms of error.

Would you have any insights about the heatmap interpretation?

Thank you once again.

AtsunoriFujita commented 2 years ago

@tjoliveira Perhap @jacobgil is pointing at BinaryClassifierOutputTarget. Because he added this before the post.