search

facebookresearch / suncet

Code to reproduce the results in the FAIR research papers "Semi-Supervised Learning of Visual Features by Non-Parametrically Predicting View Assignments with Support Samples" https://arxiv.org/abs/2104.13963 and "Supervision Accelerates Pre-training in Contrastive Semi-Supervised Learning of Visual Representations" https://arxiv.org/abs/2006.10803
MIT License
488 stars 67 forks source link

Why isnt the anchor_supports detached #12

Closed Ir1d closed 3 years ago

Ir1d commented 3 years ago

Hi, I got confused when I noticed that here the anchor_supports was directly used and not detached. I thought we should detach the results of the supports so we pull the views closer to them. By not detaching the anchor_supports, does it mean that we pull everything closer?

Thanks in advance.

MidoAssran commented 3 years ago

Hi @Ir1d, yes that's right! We don't detach the supports, which means we backpropagate with respect to the support samples as well. If you think of the support representations in a given iteration as a kind of "external memory," the idea is that you want multiple views of an image to activate similar elements in memory. Backpropagating with respect to the support is like updating your memory to make this task easier, which can lead to faster convergence.

Here is another way to possibly understand backpropagating with respect to the support: if you had a ground truth label for your anchor and used it as a target, then you essentially get a supervised contrastive loss, which you would normally differentiate with respect to both the anchors and the supports. (However, note that we never use ground truth labels as targets, and instead always use the predictions from another view as a target).