issues
search
long8v
/
PTIR
Paper Today I Read
19
stars
0
forks
source link
Paper to read
#21
Closed
long8v
closed
2 years ago
long8v
commented
2 years ago
What Does PE learn
https://arxiv.org/pdf/2010.04903.pdf
[ ]
BeiT :
https://arxiv.org/abs/2106.08254
[ ]
Masked Autoencoders Are Scalable Vision Learners :
https://arxiv.org/abs/2111.06377
[ ] PE in transformer
https://arxiv.org/pdf/2102.11090.pdf
General / 고전
[x]
트랜스포머에서 softmax 말고 relu
:
https://arxiv.org/abs/2202.08791
[x]
prenorm vs post norm in Transformer
:
https://arxiv.org/pdf/1910.05895.pdf
[x]
DALL-E :
https://arxiv.org/abs/1909.02950
) #11
[ ]
MMBT
:
https://arxiv.org/abs/1909.02950[](https://arxiv.org/pdf/2010.16402.pdf)
[ ]
언제 transfer learning을 해야 잘되나? :
https://arxiv.org/pdf/2010.16402.pdf[](https://arxiv.org/pdf/1506.03134.pdf)
[ ]
Pointer Networks
:
https://arxiv.org/pdf/1506.03134.pdf[](https://developers-shack.tistory.com/8)
[ ]
Diffusion model :
https://developers-shack.tistory.com/8[](https://arxiv.org/abs/2110.02037)
[ ]
Autoregressive Diffusion Model :
https://arxiv.org/abs/2110.02037[](https://arxiv.org/abs/1904.10509)
[ ]
Sparse Transformer :
https://arxiv.org/abs/1904.10509[](http://yann.lecun.com/exdb/publis/pdf/hadsell-chopra-lecun-06.pdf)
[ ]
Dimensionality Reduction by Learning an Invariant Mapping :
http://yann.lecun.com/exdb/publis/pdf/hadsell-chopra-lecun-06.pdf[](https://arxiv.org/abs/2104.12763)
[ ]
MDETR :
https://arxiv.org/abs/2104.12763[](https://paperswithcode.com/paper/diffusion-models-beat-gans-on-image-synthesis)
[ ]
ADM (생성)
:
https://paperswithcode.com/paper/diffusion-models-beat-gans-on-image-synthesis[](https://engineering-ladder.tistory.com/95)
[ ]
MAML :
https://engineering-ladder.tistory.com/95[](https://arxiv.org/abs/2104.12763)
MoE
[x] basic MoE :
https://arxiv.org/pdf/1701.06538.pdf
[x] DeMix :
https://arxiv.org/abs/2108.05036
[x] multi-task MoE :
https://dl.acm.org/doi/pdf/10.1145/3219819.3220007
[ ] Mixture-of-Experts with Expert Choice Routing :
https://arxiv.org/pdf/2202.09368.pdf
[x] MoEBERT :
https://arxiv.org/pdf/2204.07675.pdf
,
https://github.com/SimiaoZuo/MoEBERT
-> Distilled switch Transformer
[ ] Hash layer for Transformer :
https://arxiv.org/pdf/2106.04426.pdf
[ ] Routing Transformer :
https://arxiv.org/pdf/2003.05997.pdf
[ ] stableMoE :
https://arxiv.org/abs/2204.08396
[ ] DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learning :
https://arxiv.org/abs/2106.03760
[x] Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
[ ] Sparsely Activated Mixture-of-Experts are Robust Multi-Task Learners :
https://arxiv.org/abs/2204.07689
[ ] More Efficient In-Context Learning with GLaM :
https://ai.googleblog.com/2021/12/more-efficient-in-context-learning-with.html
[ ] Lifting the Curse of Multilinguality by Pre-training Modular Transformers :
https://arxiv.org/pdf/2205.06266.pdf
[ ] feature grouping
https://aclanthology.org/2020.coling-main.607.pdf
multi-modal
[ ]
Perceiver
:
https://arxiv.org/abs/2103.03206
[ ]
Flamingo
:
https://github.com/lucidrains/flamingo-pytorch
[ ] ALIGN
vision
[x]
BYOL :
https://arxiv.org/abs/2006.07733
#25
[x]
DINO
:
https://arxiv.org/abs/2104.14294
#26
[ ] Normalizer-Free ResNets
[ ]
ImageBART :
https://arxiv.org/pdf/2108.08827.pdf[](https://arxiv.org/abs/2106.08254)
[ ]
spatial transform networks
:
https://proceedings.neurips.cc/paper/2015/file/33ceb07bf4eeb3da587e268d663aba1a-Paper.pdf
[ ] SSL with Swin Transformer :
https://arxiv.org/pdf/2105.04553.pdf
[ ] MoCo: Momentum Contrast for Unsupervised Visual Representation Learning(simCLR 나오기전에 잘나가던 거. FAIR)
[ ] ViT 특성연구 :
https://openreview.net/pdf?id=o2mbl-Hmfgd
[ ] Few-shot class incremental learning :
https://researchmgt.monash.edu/ws/portalfiles/portal/369967836/361606468_oa.pdf
https://arxiv.org/pdf/2103.04059.pdf
[ ]
metaformer (self-attention 대체) :
https://arxiv.org/abs/2111.11418[](https://arxiv.org/pdf/2105.01601.pdf)
[ ]
MLP mixer :
https://arxiv.org/pdf/2105.01601.pdf[](https://arxiv.org/abs/1709.01507)
[ ]
Squeeze-and-Extraction
:
https://arxiv.org/abs/1709.01507[](https://arxiv.org/pdf/2105.08054.pdf)
[ ]
divide and constrast
:
https://arxiv.org/pdf/2105.08054.pdf[](https://arxiv.org/pdf/2002.05709.pdf)
[ ]
contrastive learning for image
:
https://arxiv.org/pdf/2002.05709.pdf[](https://arxiv.org/abs/2006.07733)
[ ]
VOLO
:
https://arxiv.org/pdf/2106.13112.pdf[](https://arxiv.org/pdf/2108.08827.pdf)
NLP
[ ]
ChinChileer
:
https://arxiv.org/abs/2203.15556
[ ]
colBERT
: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT :
https://arxiv.org/abs/2004.12832
[ ]
End2End MemoryNetwork
:
https://arxiv.org/pdf/1503.08895.pdf
[ ]
spanBERT
:
https://arxiv.org/abs/1907.10529[](https://paperswithcode.com/paper/a-contrastive-framework-for-neural-text)
[ ]
decoding 할때 스스로 너무 cosine 높지 않도록 제약 주기 ? :
https://paperswithcode.com/paper/a-contrastive-framework-for-neural-text[](https://arxiv.org/abs/1910.01108)
[ ]
DistilBERT :
https://arxiv.org/abs/1910.01108[](https://arxiv.org/abs/2004.05150)
[ ]
LongFormer
:
https://arxiv.org/abs/2004.05150[](https://arxiv.org/pdf/2112.04426.pdf)
[ ]
RETRO
:
https://arxiv.org/pdf/2112.04426.pdf[](https://arxiv.org/pdf/2105.03075.pdf)
[ ]
NLP augmentation survey :
https://arxiv.org/pdf/2105.03075.pdf[](https://aclanthology.org/2020.findings-emnlp.90.pdf)
[ ]
GDAUG
:
https://aclanthology.org/2020.findings-emnlp.90.pdf[](https://arxiv.org/abs/2001.04063)
[ ]
ProphetNet
:
https://arxiv.org/abs/2001.04063[](https://proceedings.neurips.cc/paper/2019/file/c04c19c2c2474dbf5f7ac4372c5b9af1-Paper.pdf)
[ ]
Cross-Lingual :
https://proceedings.neurips.cc/paper/2019/file/c04c19c2c2474dbf5f7ac4372c5b9af1-Paper.pdf[](https://arxiv.org/abs/2202.07765)
[ ]
XL :
https://arxiv.org/abs/2202.07765[](https://openreview.net/forum?id=2bO2x8NAIMB)
Document
[ ]
DiT: Self-supervised Pre-training for Document Image Transformer
:
https://arxiv.org/abs/2203.02378
[ ]
UniDoc :
https://proceedings.neurips.cc/paper/2021/file/0084ae4bc24c0795d1e6a4f58444d39b-Paper.pdf[](https://arxiv.org/abs/2201.02647)
[ ]
Data-Efficient Information Extraction :
https://arxiv.org/abs/2201.02647[](https://openaccess.thecvf.com/content/ICCV2021/papers/Appalaraju_DocFormer_End-to-End_Transformer_for_Document_Understanding_ICCV_2021_paper.pdf)
[ ]
DocFormer :
https://openaccess.thecvf.com/content/ICCV2021/papers/Appalaraju_DocFormer_End-to-End_Transformer_for_Document_Understanding_ICCV_2021_paper.pdf[](https://arxiv.org/pdf/2108.02923.pdf)
[ ]
StrucText :
https://arxiv.org/pdf/2108.02923.pdf[](https://arxiv.org/pdf/2005.13118.pdf)
long8v
commented
2 years ago
class incremental learning
Replay methods
옛날 데이터들을 1% 정도 , external memory로 저장해서
Pseudo Rehearsal : 과거 샘플들을 generate
Regularization-based methods
파라미터가 너무 바뀌지 않도록 제약을 줌
Parameter isolation
각 class seg 별로 모델을 학습하고 그 파라미터들을 어떻게 합칠지 나중에 고민
long8v
commented
2 years ago
CVPR 2022
[ ] PatchFormer: An Efficient Point Transformer with Patch Attention
[ ] AdaViT: Adaptive Vision Transformers for Efficient Image Recognition
[ ] Auto-Encoder is All You Need
[ ] Mixed Differential Privacy in Computer Vision
[ ] Self-Supervised Dense Consistency Regularization for Image-to-Image Translation
[ ] AlignMixup: Improving Representations By Interpolating Aligned Features
[ ] Distinguishing Unseen from Seen for Generalized Zero-shot Learning
[ ] Restormer: Efficient Transformer for High-Resolution Image Restoration
[ ] Few-shot Learning with Noisy Labels
[ ] Video Swin Transformer
[ ] Scaling Vision Transformers
[ ] Not Just Selection, but Exploration: Online Class-Incremental Continual Learning via Dual View Consistency
[ ] Knowledge distillation: A good teacher is patient and consistent
General / 고전
MoE
multi-modal
vision
NLP
Document