Paper to read - Githubissues

long8v commented 2 years ago

What Does PE learn https://arxiv.org/pdf/2010.04903.pdf
[ ] BeiT : https://arxiv.org/abs/2106.08254
[ ] Masked Autoencoders Are Scalable Vision Learners : https://arxiv.org/abs/2111.06377
[ ] PE in transformer https://arxiv.org/pdf/2102.11090.pdf

General / 고전

[x] 트랜스포머에서 softmax 말고 relu : https://arxiv.org/abs/2202.08791
[x] prenorm vs post norm in Transformer : https://arxiv.org/pdf/1910.05895.pdf
[x] DALL-E : https://arxiv.org/abs/1909.02950) #11
[ ] MMBT : https://arxiv.org/abs/1909.02950[](https://arxiv.org/pdf/2010.16402.pdf)
[ ] 언제 transfer learning을 해야 잘되나? : https://arxiv.org/pdf/2010.16402.pdf[](https://arxiv.org/pdf/1506.03134.pdf)
[ ] Pointer Networks : https://arxiv.org/pdf/1506.03134.pdf[](https://developers-shack.tistory.com/8)
[ ] Diffusion model : https://developers-shack.tistory.com/8[](https://arxiv.org/abs/2110.02037)
[ ] Autoregressive Diffusion Model : https://arxiv.org/abs/2110.02037[](https://arxiv.org/abs/1904.10509)
[ ] Sparse Transformer : https://arxiv.org/abs/1904.10509[](http://yann.lecun.com/exdb/publis/pdf/hadsell-chopra-lecun-06.pdf)
[ ] Dimensionality Reduction by Learning an Invariant Mapping : http://yann.lecun.com/exdb/publis/pdf/hadsell-chopra-lecun-06.pdf[](https://arxiv.org/abs/2104.12763)
[ ] MDETR : https://arxiv.org/abs/2104.12763[](https://paperswithcode.com/paper/diffusion-models-beat-gans-on-image-synthesis)
[ ] ADM (생성) : https://paperswithcode.com/paper/diffusion-models-beat-gans-on-image-synthesis[](https://engineering-ladder.tistory.com/95)
[ ] MAML : https://engineering-ladder.tistory.com/95[](https://arxiv.org/abs/2104.12763)

MoE

[x] basic MoE : https://arxiv.org/pdf/1701.06538.pdf
[x] DeMix : https://arxiv.org/abs/2108.05036
[x] multi-task MoE : https://dl.acm.org/doi/pdf/10.1145/3219819.3220007
[ ] Mixture-of-Experts with Expert Choice Routing : https://arxiv.org/pdf/2202.09368.pdf
[x] MoEBERT : https://arxiv.org/pdf/2204.07675.pdf, https://github.com/SimiaoZuo/MoEBERT -> Distilled switch Transformer
- [ ] Hash layer for Transformer : https://arxiv.org/pdf/2106.04426.pdf
- [ ] Routing Transformer : https://arxiv.org/pdf/2003.05997.pdf
[ ] stableMoE : https://arxiv.org/abs/2204.08396
[ ] DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learning : https://arxiv.org/abs/2106.03760
[x] Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
[ ] Sparsely Activated Mixture-of-Experts are Robust Multi-Task Learners : https://arxiv.org/abs/2204.07689
[ ] More Efficient In-Context Learning with GLaM : https://ai.googleblog.com/2021/12/more-efficient-in-context-learning-with.html
[ ] Lifting the Curse of Multilinguality by Pre-training Modular Transformers : https://arxiv.org/pdf/2205.06266.pdf
[ ] feature grouping https://aclanthology.org/2020.coling-main.607.pdf

multi-modal

[ ] Perceiver : https://arxiv.org/abs/2103.03206
[ ] Flamingo : https://github.com/lucidrains/flamingo-pytorch
[ ] ALIGN

vision

[x] BYOL : https://arxiv.org/abs/2006.07733 #25
[x] DINO : https://arxiv.org/abs/2104.14294 #26
[ ] Normalizer-Free ResNets
[ ] ImageBART : https://arxiv.org/pdf/2108.08827.pdf[](https://arxiv.org/abs/2106.08254)
[ ] spatial transform networks : https://proceedings.neurips.cc/paper/2015/file/33ceb07bf4eeb3da587e268d663aba1a-Paper.pdf
[ ] SSL with Swin Transformer : https://arxiv.org/pdf/2105.04553.pdf
[ ] MoCo: Momentum Contrast for Unsupervised Visual Representation Learning(simCLR 나오기전에 잘나가던 거. FAIR)
[ ] ViT 특성연구 : https://openreview.net/pdf?id=o2mbl-Hmfgd
[ ] Few-shot class incremental learning : https://researchmgt.monash.edu/ws/portalfiles/portal/369967836/361606468_oa.pdf https://arxiv.org/pdf/2103.04059.pdf
[ ] metaformer (self-attention 대체) : https://arxiv.org/abs/2111.11418[](https://arxiv.org/pdf/2105.01601.pdf)
[ ] MLP mixer : https://arxiv.org/pdf/2105.01601.pdf[](https://arxiv.org/abs/1709.01507)
[ ] Squeeze-and-Extraction : https://arxiv.org/abs/1709.01507[](https://arxiv.org/pdf/2105.08054.pdf)
[ ] divide and constrast : https://arxiv.org/pdf/2105.08054.pdf[](https://arxiv.org/pdf/2002.05709.pdf)
[ ] contrastive learning for image : https://arxiv.org/pdf/2002.05709.pdf[](https://arxiv.org/abs/2006.07733)
[ ] VOLO : https://arxiv.org/pdf/2106.13112.pdf[](https://arxiv.org/pdf/2108.08827.pdf)
NLP
[ ] ChinChileer : https://arxiv.org/abs/2203.15556
[ ] colBERT : Efficient and Effective Passage Search via Contextualized Late Interaction over BERT : https://arxiv.org/abs/2004.12832
[ ] End2End MemoryNetwork : https://arxiv.org/pdf/1503.08895.pdf
[ ] spanBERT : https://arxiv.org/abs/1907.10529[](https://paperswithcode.com/paper/a-contrastive-framework-for-neural-text)
[ ] decoding 할때 스스로 너무 cosine 높지 않도록 제약 주기 ? : https://paperswithcode.com/paper/a-contrastive-framework-for-neural-text[](https://arxiv.org/abs/1910.01108)
[ ] DistilBERT : https://arxiv.org/abs/1910.01108[](https://arxiv.org/abs/2004.05150)
[ ] LongFormer : https://arxiv.org/abs/2004.05150[](https://arxiv.org/pdf/2112.04426.pdf)
[ ] RETRO : https://arxiv.org/pdf/2112.04426.pdf[](https://arxiv.org/pdf/2105.03075.pdf)
[ ] NLP augmentation survey : https://arxiv.org/pdf/2105.03075.pdf[](https://aclanthology.org/2020.findings-emnlp.90.pdf)
[ ] GDAUG: https://aclanthology.org/2020.findings-emnlp.90.pdf[](https://arxiv.org/abs/2001.04063)
[ ] ProphetNet : https://arxiv.org/abs/2001.04063[](https://proceedings.neurips.cc/paper/2019/file/c04c19c2c2474dbf5f7ac4372c5b9af1-Paper.pdf)
[ ] Cross-Lingual : https://proceedings.neurips.cc/paper/2019/file/c04c19c2c2474dbf5f7ac4372c5b9af1-Paper.pdf[](https://arxiv.org/abs/2202.07765)
[ ] XL : https://arxiv.org/abs/2202.07765[](https://openreview.net/forum?id=2bO2x8NAIMB)

Document

[ ] DiT: Self-supervised Pre-training for Document Image Transformer : https://arxiv.org/abs/2203.02378
[ ] UniDoc : https://proceedings.neurips.cc/paper/2021/file/0084ae4bc24c0795d1e6a4f58444d39b-Paper.pdf[](https://arxiv.org/abs/2201.02647)
[ ] Data-Efficient Information Extraction : https://arxiv.org/abs/2201.02647[](https://openaccess.thecvf.com/content/ICCV2021/papers/Appalaraju_DocFormer_End-to-End_Transformer_for_Document_Understanding_ICCV_2021_paper.pdf)
[ ] DocFormer : https://openaccess.thecvf.com/content/ICCV2021/papers/Appalaraju_DocFormer_End-to-End_Transformer_for_Document_Understanding_ICCV_2021_paper.pdf[](https://arxiv.org/pdf/2108.02923.pdf)
[ ] StrucText : https://arxiv.org/pdf/2108.02923.pdf[](https://arxiv.org/pdf/2005.13118.pdf)

long8v commented 2 years ago

class incremental learning

Replay methods

옛날 데이터들을 1% 정도 , external memory로 저장해서
Pseudo Rehearsal : 과거 샘플들을 generate

Regularization-based methods

파라미터가 너무 바뀌지 않도록 제약을 줌

Parameter isolation

각 class seg 별로 모델을 학습하고 그 파라미터들을 어떻게 합칠지 나중에 고민

long8v commented 2 years ago

CVPR 2022

[ ] PatchFormer: An Efficient Point Transformer with Patch Attention
[ ] AdaViT: Adaptive Vision Transformers for Efficient Image Recognition
[ ] Auto-Encoder is All You Need
[ ] Mixed Differential Privacy in Computer Vision
[ ] Self-Supervised Dense Consistency Regularization for Image-to-Image Translation
[ ] AlignMixup: Improving Representations By Interpolating Aligned Features
[ ] Distinguishing Unseen from Seen for Generalized Zero-shot Learning
[ ] Restormer: Efficient Transformer for High-Resolution Image Restoration
[ ] Few-shot Learning with Noisy Labels
[ ] Video Swin Transformer
[ ] Scaling Vision Transformers
[ ] Not Just Selection, but Exploration: Online Class-Incremental Continual Learning via Dual View Consistency
[ ] Knowledge distillation: A good teacher is patient and consistent

long8v / PTIR

Paper to read #21

General / 고전

MoE

multi-modal

vision

NLP

Document

class incremental learning

CVPR 2022