search

yitu-opensource / T2T-ViT

ICCV2021, Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet
Other
1.14k stars 177 forks source link

Pre-train ViT on unsupervised task? #43

Closed neel04 closed 3 years ago

neel04 commented 3 years ago

Thanks a lot for this repo! I am looking to pre-train ViT in an unsupervised way, but it doesn't seem like it's possible with this repo.

Is there any way I could do that? would you happen to know any other method/repo where we can pre-train it in an unsupervised manner (i.e on a dataset not labeled)?

do you think that specifying a dummy class and pre-training that way may provide extra accuracy when fine-tuning on the labeled dataset?

Many thanks :hugs:

yuanli2333 commented 3 years ago

Hi, if you want to know more about self-supervised learning by vision transformer, I recommend this paper