Attention-viz with new models and datasets?

catherinesyeh / attention-viz

Visualizing query-key interactions in language + vision transformers

http://attentionviz.com/

MIT License

122 stars 15 forks source link

Attention-viz with new models and datasets? #72

Closed dbl001 closed 5 months ago

dbl001 commented 9 months ago

Very nice tool! I would like to use AttentionViz on other models and datasets.

I am working with Karpaty's Llama2.c project and a dataset of Covid-19 research papers.

Do you have a tool which generates the: tokens.json agg_attension.json layer{n}_attention{m}.json

from a trained model along with the input sentences?

https://github.com/karpathy/llama2.c https://allenai.org/data/cord-19

catherinesyeh commented 5 months ago

Thank you so much for checking out AttentionViz and apologies for missing this issue until now!

Feel free to check out these notebooks: generate_data.ipynb and generate_vit_data.ipynb. We are not actively maintaining these, however, and you may need to tweak them slightly to fit your needs. But I hope they will be helpful in showing how to generate similar data files!