search

facebookresearch / suncet

Code to reproduce the results in the FAIR research papers "Semi-Supervised Learning of Visual Features by Non-Parametrically Predicting View Assignments with Support Samples" https://arxiv.org/abs/2104.13963 and "Supervision Accelerates Pre-training in Contrastive Semi-Supervised Learning of Visual Representations" https://arxiv.org/abs/2006.10803
MIT License
486 stars 67 forks source link

"training" parameter missing from "meta" section of config file cifar10_snn.yaml #23

Closed priyamdey closed 3 years ago

priyamdey commented 3 years ago

Hi! I was trying to finetune a pretrained model for cifar10 subsets. Regarding this, I have got 2 questions:

  1. In the file "src/fine_tune.py", we are importing "training" parameter from "meta" section of args (line no. 65). But this parameter is missing from the config file "configs/paws/cifar10_snn.yaml". Am I referring to the wrong config file here?
  2. Why have we kept training=True while creating validation dataset of cifar10 (line no. 154 in src/fine_tune.py)?

Thanks!

MidoAssran commented 3 years ago

Hi @priyamdey,

  1. The config file is correct, but the cifar10 models are fine-tuned with src/snn_fine_tune.py and evaluated with snn_eval.py
  2. This is just a subtlety of the variable naming, since the validation set is a subset of the train set. This line here tells the model to use the validation set, not the train set.
priyamdey commented 3 years ago

Ah ok. Thanks for the clarification!

priyamdey commented 3 years ago

Sorry, maybe I got a bit confused here. In the appendix of the paper, under the section of "Fine-tuning details", the following is mentioned: Following [1], we fine-tune a linear classifier from the first layer of the projection head in the pretrained encoder fθ, and initialize the weights of the linear classifier to zero. Specifically, we simultaneously fine-tune the encoder/classifier weights by optimizing a supervised cross-entropy loss on the small set of available labeled samples.

As you mentioned out that for fine-tuning, src/snn_fine_tune.py script is used. However, I see suncet loss and 3-layer MLP as the projection head being used in that script. src/fine_tune.py has the right loss (CE) and MLP (1-layer followed by linear clf) based on what is mentioned in the appendix.

MidoAssran commented 3 years ago

Yes that's true for ImageNet. But for CIFAR10 we just do nearest-neighbours classification/fine-tuning (see appendix C)

priyamdey commented 3 years ago

Oh I see. I missed that. Now it's clear. Thank you for pointing that out!