search

hila-chefer / Transformer-MM-Explainability

[ICCV 2021- Oral] Official PyTorch implementation for Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers, a novel method to visualize any Transformer-based network. Including examples for DETR, VQA.
MIT License
801 stars 107 forks source link

Object detection/Segmentation Explainability #19

Closed jaiswati closed 2 years ago

jaiswati commented 2 years ago

Hello @hila-chefer,

How to get the relevance maps with Swin Transformer for Object detection/Segmentation using this technique?

Best, @jaiswati

hila-chefer commented 2 years ago

Hi @jaiswati, thanks for your interest!

Our method does not support hierarchical models such as Swin out of the box. It would require some non-trivial modifications, that we have not yet attempted. I’ll update this issue if we adapt our method to Swin as well.

Best, Hila.