Why cos_sim after L2 norm?

sthalles / SimCLR

PyTorch implementation of SimCLR: A Simple Framework for Contrastive Learning of Visual Representations

https://sthalles.github.io/simple-self-supervised-learning/

MIT License

2.21k stars 458 forks source link

Why cos_sim after L2 norm? #14

Closed BoPang1996 closed 3 years ago

BoPang1996 commented 4 years ago

Hi, This code is really useful for me. Thanks! But I got a question about the NT-Xent loss. I noticed that you use L2 norm on z and then use cos_similarity after that. But cos_similarity already contain the function of l2 norm. Why use L2 norm first?

nzw0301 commented 4 years ago

@BoPang1996 I think you are right. In this repo with default config parameters, z is normalized twice. So, we can set False to this config parameter or remove the first normalization part on yours to make the training faster.

sthalles commented 3 years ago

Hello guys, that is a general message to say that I have refactored the whole project. I believe the project is much easier to understand now. Please have a look at the new impl and free to submit PR if you find any bugs. Thanks.