Implement VICREg

[QuentinDuval]

🌟 New SSL approach addition

Approach description

Recent self-supervised methods for image representation learning are based on maximizing the agreement between embedding vectors from different views of the same image. A trivial solution is obtained when the encoder outputs constant vectors. This collapse problem is often avoided through implicit biases in the learning architecture, that often lack a clear justification or interpretation. In this paper, we introduce VICReg (Variance-Invariance-Covariance Regularization), a method that explicitly avoids the collapse problem with a simple regularization term on the variance of the embeddings along each dimension individually. VICReg combines the variance term with a decorrelation mechanism based on redundancy reduction and covariance regularization, and achieves results on par with the state of the art on several downstream tasks. In addition, we show that incorporating our new variance term into other methods helps stabilize the training and leads to performance improvements.

https://arxiv.org/pdf/2105.04906.pdf

Architecture

Open source status

• [ ] the model implementation is available: (give details)
• [ ] the model weights are available: (give details)
• who are the authors: Adrien Bardes (@Adrien987k), Jean Ponce, Yann LeCun

[clane9]

Hi @QuentinDuval, I saw that the official code was recently released. I'd love to see the method added to VISSL as well. Are there plans to add it in progress? If not, I'd be happy to open a PR. Let me know, thanks!

[crypdick]

@clane9 I would love to see vicreg supported in vissl

[crypdick]

@QuentinDuval is there an update on this? I'm keen to try out VICREG.

[clane9]

Hi @crypdick, @QuentinDuval. I did have a draft implementation of VICReg in my fork. I'd have to rebase and finish it up, but then I'd be happy to open a PR. lmk!