search

zhoudaquan / dvit_repo

MIT License
136 stars 23 forks source link

Attention map visualization #5

Open wofmanaf opened 3 years ago

wofmanaf commented 3 years ago

I notice that you visualize the attention map of selected blocks(in Fig. 6), can you show the code for drawing that?

zhoudaquan commented 3 years ago

I notice that you visualize the attention map of selected blocks(in Fig. 6), can you show the code for drawing that?

Hi, Sorry for this late reply. I am doing another project recently. I take the average of the attention map over the head dim and then visualize the attention map with Seaborn. I can upload it later after some simple clearning.

xwan6266 commented 3 years ago

Thanks for this nice work. I'm also interested with your attention map visualisation, did you use Seaborn heatmap to visualise this after training? I tried to visualise my fully trained attention weights but the outcome failed to show similar texture like the Figure 6 shows in the paper. Could you please share some details of your implementation?

zhoudaquan commented 3 years ago

Thanks for this nice work. I'm also interested with your attention map visualisation, did you use Seaborn heatmap to visualise this after training? I tried to visualise my fully trained attention weights but the outcome failed to show similar texture like the Figure 6 shows in the paper. Could you please share some details of your implementation?

Hi, thanks for your interest and sorry for the delay on uploading the visualization script. I am doing some other projects. Just a quick reply, I take the average over all heads before visualization and yes, I use seaborn for visualization. With default setting, there are only some white lines along the diagonal and the range keeps increasing with a diagonal center. The more clear visualization need to adjust the contrast settings carefully. Can you please show some examples of your plots? I can help with debugging a little bit hopefully. Later I can also upload mine for comparisons.

xwan6266 commented 3 years ago

Thanks for this nice work. I'm also interested with your attention map visualisation, did you use Seaborn heatmap to visualise this after training? I tried to visualise my fully trained attention weights but the outcome failed to show similar texture like the Figure 6 shows in the paper. Could you please share some details of your implementation?

Hi, thanks for your interest and sorry for the delay on uploading the visualization script. I am doing some other projects. Just a quick reply, I take the average over all heads before visualization and yes, I use seaborn for visualization. With default setting, there are only some white lines along the diagonal and the range keeps increasing with a diagonal center. The more clear visualization need to adjust the contrast settings carefully. Can you please show some examples of your plots? I can help with debugging a little bit hopefully. Later I can also upload mine for comparisons.

Dear Daquan, Thanks a lot for your quick reply. I finally managed to make it work similarly as yours. It turns out that I was choosing inappropriate colour during visualisation, and maybe not fully trained model for visualisation.

cenchaojun commented 2 years ago

I notice that you visualize the attention map of selected blocks(in Fig. 6), can you show the code for drawing that?

Hi, Sorry for this late reply. I am doing another project recently. I take the average of the attention map over the head dim and then visualize the attention map with Seaborn. I can upload it later after some simple clearning. Dear zhoudaquan Do you upload the code of visualization feature map?

zhoudaquan commented 2 years ago

I notice that you visualize the attention map of selected blocks(in Fig. 6), can you show the code for drawing that?

Hi, Sorry for this late reply. I am doing another project recently. I take the average of the attention map over the head dim and then visualize the attention map with Seaborn. I can upload it later after some simple clearning. Dear zhoudaquan Do you upload the code of visualization feature map?

Hi Chaojun, Thanks for your interest and really sorry for this really late reply! You can take a look at this link first where I have uploaded the code for visualization as well as the pretrained model and the saved attention map data in pickle file. I will try to update the README this weekend.

https://drive.google.com/drive/folders/1_lxspG_nzPstxDWhKQqPWhYZlB6zPMGs?usp=sharing

zhoudaquan commented 2 years ago

I notice that you visualize the attention map of selected blocks(in Fig. 6), can you show the code for drawing that?

Hi,

thanks for your interest. Please check the link for the visualizations.

https://drive.google.com/drive/folders/1_lxspG_nzPstxDWhKQqPWhYZlB6zPMGs?usp=sharing

cenchaojun commented 2 years ago

I notice that you visualize the attention map of selected blocks(in Fig. 6), can you show the code for drawing that?

Hi, Sorry for this late reply. I am doing another project recently. I take the average of the attention map over the head dim and then visualize the attention map with Seaborn. I can upload it later after some simple clearning. Dear zhoudaquan Do you upload the code of visualization feature map?

Hi Chaojun, Thanks for your interest and really sorry for this really late reply! You can take a look at this link first where I have uploaded the code for visualization as well as the pretrained model and the saved attention map data in pickle file. I will try to update the README this weekend.

https://drive.google.com/drive/folders/1_lxspG_nzPstxDWhKQqPWhYZlB6zPMGs?usp=sharing

thank you so much, this code will help me a lot

ychengrong commented 2 years ago

I notice that you visualize the attention map of selected blocks(in Fig. 6), can you show the code for drawing that?

Hi,

thanks for your interest. Please check the link for the visualizations.

https://drive.google.com/drive/folders/1_lxspG_nzPstxDWhKQqPWhYZlB6zPMGs?usp=sharing

Hi, Daquan! I tried the code and .pth.tar file you provided above. However, I got the output visualization for layer 1 like this. image The key to the model I used was "blocks.{layer_index}.attn.qkv.weight". Can you give me some advice about this? Appreciate that!

zhoudaquan commented 2 years ago

I notice that you visualize the attention map of selected blocks(in Fig. 6), can you show the code for drawing that?

Hi, thanks for your interest. Please check the link for the visualizations. https://drive.google.com/drive/folders/1_lxspG_nzPstxDWhKQqPWhYZlB6zPMGs?usp=sharing

Hi, Daquan! I tried the code and .pth.tar file you provided above. However, I got the output visualization for layer 1 like this. image The key to the model I used was "blocks.{layer_index}.attn.qkv.weight". Can you give me some advice about this? Appreciate that!

Please see my reply in another thread. Thanks.