How can we visualize the self-attention map

pbloem / former

Simple transformer implementation from scratch in pytorch.

http://peterbloem.nl/blog/transformers

MIT License

1.03k stars 170 forks source link

Closed amiltonwong closed 4 years ago

amiltonwong commented 4 years ago

Hi, @pbloem ,

How can we visualize the self-attention map? Could you provide the corresponding code snippet?

Thanks!

pbloem commented 4 years ago

The attention map is the tensor called dot in the code. You could feed it to a function like matplotlib's imshow.

I hope that points you in the right direction. You'll have to write the code yourself, I'm afraid.