Tutorial to compute Transformer FLOPs (Equation 6 in the paper)

yuweihao / MambaOut

MambaOut: Do We Really Need Mamba for Vision?

Apache License 2.0

1.95k stars 31 forks source link

Tutorial to compute Transformer FLOPs (Equation 6 in the paper) #210

Open yuweihao opened 3 months ago

yuweihao commented 3 months ago

Hi everyone, Someone asked me how to compute Transformer FLOPs (Equation 6 in the paper). I wrote a tutorial and hope it can help. Feel free to offer feedback so that I can improve it. transformer_FLOPs

You can also refer to PDF version.

WowLucy commented 3 months ago

Thank you very much! May I ask how you draw those colored squares for matrices? It looks like latex, but I could not find related latex code online. Thank you again!

yuweihao commented 3 months ago

Hi @WowLucy, I used draw.io and just clumsily assembled the small squares into a matrix.

WowLucy commented 3 months ago

Cool. Thank you!

Yubel426 commented 1 month ago

Hi, the tutorial looks great! What tool did you use to create this PDF?

yuweihao commented 1 month ago

Hi @Yubel426 , thank you! I used draw.io to create it.