junxnone / aiwiki

AI Wiki

https://junxnone.github.io/aiwiki

18 stars 2 forks source link

ML Transformer EfficientAttention #225

Open junxnone opened 3 years ago

junxnone commented 3 years ago

Efficient Attention

Reference

Efficient Attention
2020-09 Efficient Transformers: A Survey [Paper] [Note]
Transformers大家族——Efficient Transformers: A Survey

Brief

Efficient Transformer [打算解决的问题]
- 减小内存占用
- 减少计算量
- 增加长序列数据的处理能力
Dense Attention & Sparse Attention

Name	Description

Reformer Longformer Linformer Performer Linear Transformers Big Bird

junxnone commented 3 years ago

junxnone/aiwiki#212