-
Implement the linformer model from the following paper: https://arxiv.org/pdf/2006.04768.pdf
The implementation will be based on the opensource implementation: https://github.com/tatp22/linformer-pyt…
-
## Linformer addition thoughts 🤷🏻♂️
Hey I just noticed that a linear transform was just created! I think it would be interesting to add it in DETR since we are doing image detection our sequences …
-
Thanks a lot for adding the official code for Linformer to FairSeq! Are you also planning on releasing some pre-trained weights for the model?
-
The [Linformer](https://arxiv.org/abs/2006.04768v2) layer looks very interesting w.r.t. managing computational complexity of transformers. Having such an implementation would prove useful for upcoming…
-
Hi. Thanks for the wonderful implementation!
I was wondering if linformer can be used with any unordered set of tensors (or is it just sequence data?). Specifically, is linformer permutation equiva…
-
# Transformer 在视觉方面的应用
## Reference
- 2021-01 A Survey on Visual Transformer [[Paper](https://arxiv.org/pdf/2012.12556.pdf)] [[Note](https://github.com/junxnone/tech-io/issues/926)]
- 2021-01 Tr…
-
Hi,
Thanks for the excellent work!
I found some issues in my humble trials (I didn't change anything in the code):
1. using softmax attention on Text4k I got ~63.7 acc instead of 65.02 you pos…
-
# Efficient Attention
## Reference
- [Efficient Attention](https://github.com/Separius/awesome-fast-attention)
- 2020-09 Efficient Transformers: A Survey [[Paper](https://arxiv.org/abs/2009.06732…
-
Hello,
Sorry if this is a silly question, but looking at your code in ptr_base.py line 90 the LinformerEncoder layer doesn't seem to be implementing linear attention at all; what it seems to be doi…
-
I find a space-wise performance gain when I have long sequence with small feature dimension for sure.
But I do not find any time-wise performance gain, and under the same condition, I find loss dro…