Can you expand the lrp.py to cover BigBird?

hila-chefer / Transformer-Explainability

[CVPR 2021] Official PyTorch implementation for Transformer Interpretability Beyond Attention Visualization, a novel method to visualize classifications by Transformer based networks.

MIT License

1.75k stars 232 forks source link

Can you expand the lrp.py to cover BigBird? #43

Closed ngoel17 closed 2 years ago

ngoel17 commented 2 years ago

Most of the NLP tasks use BigBird due to long text support. Can you please extend the class to support BigBird, by adding rules for torch.nn.modules.sparse.Embedding etc?

hila-chefer commented 2 years ago

Hi @ngoel17, thank you for your interest! We currently do not have plans to extend our implementation to BigBird, however, I'd suggest using the method from our second paper which does not require LRP (only gradient propagation).

Best, Hila.