Models trained only with self supervision

[MorenoLaQuatra]

Hi,

First of all, thank you all for the amazing work on SSL. I was wondering if the models released at the following link: https://console.cloud.google.com/storage/browser/simclr-checkpoints/simclrv2/pretrained are trained without any supervision. In other words, can you confirm that those models have never seen ImageNet labels during training? If not, is there any model that is fully trained with SSL (without any ImageNet label training)?

MLQ

[chentingpc]

Yes, the pretrained models (resnet backbone and projection head) are trained only using unlabeled images. However, please note that the linear head weights (head_supervised in checkpoint) are trained with labels. In these models, linear head was trained along the unsupervised pretraining but with a stop gradient to prevent the label information leaking to the backbone and projection head. We found this produces similar results with first perform the unsupervised training of backbone and projection head, and then train the linear head on top separately.

[MorenoLaQuatra]

Thank you so much for the quick and detailed answer!