search

google-research / simclr

SimCLRv2 - Big Self-Supervised Models are Strong Semi-Supervised Learners
https://arxiv.org/abs/2006.10029
Apache License 2.0
4.08k stars 623 forks source link

Semi-supervised Learning via Fine-Tuning #27

Closed ksoy0128 closed 4 years ago

ksoy0128 commented 4 years ago

Is there code for semi-supervised learning via Fine-tuning(Zhai et al. (2019))? I can't find it. Please let me know if there is a way.

I have more questions about semi-supervised learning via fine-tuning(Zhai et al. (2019)). When fine-tuning, the network is updated using the following Loss

image

Here are some questions.

In semi-supervised learning, is the loss used for unlabeled dataset the same as the contrastive loss used in SIMCLR-based pretraining? (Augmentation-> encoder-> MLP-> contrastive loss) or Is it loss such as Loss_rot used in Zhai et al (2019)?

When fine-tuning with semi-supervised learning, do you learn after replacing g (.) (MLP) with a head that acts as a classifier? Or is there a layer for classification independent of g (.)? At this time, is it correct to fine-tun both the encoder and the changed head (ex FC layer)?

Finally, I wonder about backpropagation from contrastive loss of unlabeled data and how it can be fine-tuned not only to encoder but also to FC for classification.

chentingpc commented 4 years ago

Hi it seem you may have confused the fine-tuning in our work with Zhai et al (2019). For semi-supervised learning, we simply use fine-tuning on the labeled examples. No extra loss is added. We only fine-tune the base network (not including g), and you can use the fine-tuning script in ReadMe page to do that (the 1%/10% imagenet dataset can be access via tensorflow datasets now: https://www.tensorflow.org/datasets/catalog/imagenet2012_subset)

ksoy0128 commented 4 years ago

@chentingpc
Thank you :)

I have a question in the following fine tuning situation (Semi-supervised learning)

Training a linear classifier at the same time adjusting all weights in base network (i.e. fine-tune all layers)

1) According to the answer, when fine-tuning, only labeled data is used. If so, isn't contrastive loss used in fine-tuning? (Process such as obtaining coasine similarity)

2) I know that semi-supervised learning is used when there are many unlabeled data in the dataset and little labeled data. Why is it called semi supervised learning when fine tuning using only labeled data? (Pretraining with unlabeld data with fine tuning with small amount of labeled data => Is it semi supervised learning?)

3) Is it necessary to use self-supervised learning from scratch to classify images in a new domain with little labeled dataset?

chentingpc commented 4 years ago
  1. no contrastive loss is used during fine-tuning.
  2. it fits the concept of semi-supervised learning in the sense it can leverage unlabeled data to improve performance, but it is also different from most existing semi-supervised methods. it just uses unlabeled data in a different way (via pretraining).
  3. I think that really depends on how different the domain and how much domain specific data you have. If you have lots of unlabeled data in that domain, you could potentially benefit from pretraining on those as well.
ines321 commented 3 years ago

Can you explain to me the main difference between self-supervised and semi-supervised Leraning