[ICCV 2021- Oral] Official PyTorch implementation for Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers, a novel method to visualize any Transformer-based network. Including examples for DETR, VQA.
MIT License
801
stars
107
forks
source link
self.attn_probs in ResidualAttentionBlock() causes problems - how to make explainability work with mlfoundations / open_clip model #32
I am running experiments using OpenCLIP model from mlfoundations. . It turns out that openCLIP's architecture is different from CLIP's architecture you defined. I encountered the same error as mentioned here, then tried to add missing parts but somehow still get attn_probs = None at the end, instead of being a tensor calculated along the way.
Could you please guide me on how to make your code work with openCLIP? In particular, what should be enough to modify in respective modules of openCLIP ( transformer.py is the equivalent module to yours model.py).
P.S. if you are interested in working on this for an hourly-based compensation, I would gladly discuss it. Please let me know.
Hi,
I am running experiments using OpenCLIP model from mlfoundations. . It turns out that openCLIP's architecture is different from CLIP's architecture you defined. I encountered the same error as mentioned here, then tried to add missing parts but somehow still get attn_probs = None at the end, instead of being a tensor calculated along the way.
Could you please guide me on how to make your code work with openCLIP? In particular, what should be enough to modify in respective modules of openCLIP ( transformer.py is the equivalent module to yours model.py).
P.S. if you are interested in working on this for an hourly-based compensation, I would gladly discuss it. Please let me know.