This repository serves as a knowledge base with key insights, details from other research and implementations to serve as references and one place to document various possible paths to achieve something.
GNU General Public License v3.0
4
stars
0
forks
source link
Neko : Checkout efficient attention mechanisms for multimodality #2
The survey paper mainly suggested attention variants each of which is quadratic in computational complexity.
However, there have been many different advancements in this area from BigBird, Longformer, Sparse Transformer, Linformer, to dilated attention, all of which are better than quadratic, with some of them being linear as well.
Some investigation around these could be added to the Knowledge Base.
The survey paper mainly suggested attention variants each of which is quadratic in computational complexity.
However, there have been many different advancements in this area from BigBird, Longformer, Sparse Transformer, Linformer, to dilated attention, all of which are better than quadratic, with some of them being linear as well.
Some investigation around these could be added to the Knowledge Base.