Display sparsity of attention patterns

catherinesyeh / attention-viz

Visualizing query-key interactions in language + vision transformers

http://attentionviz.com/

MIT License

122 stars 15 forks source link

Open catherinesyeh opened 1 year ago

catherinesyeh commented 1 year ago

[ ] Show avg number of tokens each token in a sentence is paying attention to or distribution overall
[ ] Could also do same thing for each head overall