Loss function calculation

princeton-nlp / SimCSE

[EMNLP 2021] SimCSE: Simple Contrastive Learning of Sentence Embeddings https://arxiv.org/abs/2104.08821

MIT License

3.36k stars 507 forks source link

Loss function calculation #164

Closed natuhvnh closed 2 years ago

natuhvnh commented 2 years ago

Hi, I have a question about the loss function calculation: Screenshot from 2022-05-05 12-10-12 It means that we treat x(i) and x(j) similar to x(i) and its contradiction hypotheses ? If yes, the loss function will not work well in case x(i) has many entialment hypothesis in the datasets and few of those appear in the mini-batch ?

gaotianyu1350 commented 2 years ago

Hi,

In this form, xi+ is the entailment of xi. Can you elaborate more on "few of those appear in the mini-batch"?

natuhvnh commented 2 years ago

I mean that somehow hj+ is similar to hi+ (both are entailment of hi. But here hj+ is in the denominator, dose it effect the performance?

gaotianyu1350 commented 2 years ago

Hi,

Hj+ is not the entailment sentence for hi (it is the entailment for hj). It is supposed to be different from hi.