search

rayleizhu / BiFormer

[CVPR 2023] Official code release of our paper "BiFormer: Vision Transformer with Bi-Level Routing Attention"
https://arxiv.org/abs/2303.08810
MIT License
460 stars 36 forks source link

colab tutorial？ #28

Open yejy53 opened 11 months ago

yejy53 commented 11 months ago

Thank you for your analysis, can you post a colab tutorial?

rayleizhu commented 11 months ago

It is a good suggestion. However, it takes quite some effort to tidy up the visualization code and build a nice demo. I'm not sure when I will have the motivation to finish it, since I'm chasing the next CVPR deadline, and there are too many things to do.

I can give some tips for the implementation if you would like to do it by yourself:

  1. You can use register_forward_hook() to easily get the routing region indices and attention matrix, without modifying the model itself.
  2. You can use cv2.applyColorMap to convert attention score map to pseudo-colored heat map.
  3. You can then alpha blend the image and heatmap (i.e. alphaimage + (1-alpha)heatmap) to get the visualization figure.
yuxiongl commented 9 months ago

Hello author, I am a newcomer in this field. Could you tell me the specific code for using [register_forward_hook()]. The model stages are nested too deeply. It is difficult for me to figure out the layer name. I don't know how to keep pointing out the middle value like model. bla. bla. bla.