Confusions with info_nce loss in gsimclr.py

Shen-Lab / GraphCL

[NeurIPS 2020] "Graph Contrastive Learning with Augmentations" by Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, Yang Shen

MIT License

559 stars 103 forks source link

Confusions with info_nce loss in gsimclr.py #27

Open ZhuYun97 opened 3 years ago

ZhuYun97 commented 3 years ago

Thanks for your awesome codes, but I have some questions about ./unsupervised_TU/gsimclr.py file. In the loss_cal function, the denominator will minus the positive pair loss = pos_sim / (sim_matrix.sum(dim=1) - pos_sim) which is not consistent with infonce loss $\ell_{i, j}=-\log \frac{\exp \left(\operatorname{sim}\left(\boldsymbol{z}_{i}, \boldsymbol{z}_{j}\right) / \tau\right)}{\sum_{k=1}^{N} \exp \left(\operatorname{sim}\left(\boldsymbol{z}_{i}, \boldsymbol{z}_{k}\right) / \tau\right)}$

yyou1996 commented 3 years ago

Hi @ZhuYun97,

Thank you for asking. Please see the in the paper https://arxiv.org/pdf/2010.13902.pdf eq. (3), and also refer to the eq. (1) in simclr paper http://proceedings.mlr.press/v119/chen20j/chen20j.pdf.

ZhuYun97 commented 3 years ago

Thanks for your reply. But I still have some questions. The denominator contains all pairs(both positive and negative) in simclr(eq. 1). But in graphcl(eq. 3), the denominator does not contain positive pair. I am confused why the denominator of contrastive loss in graphcl doesn't contain positive pair. Does the accuracy perform better than the contrastive loss which contains positive pair in the denominator?

yyou1996 commented 3 years ago

Numerically I did not find a difference.