search

princeton-nlp / SimCSE

[EMNLP 2021] SimCSE: Simple Contrastive Learning of Sentence Embeddings https://arxiv.org/abs/2104.08821
MIT License
3.36k stars 507 forks source link

the use of hard_neg_weight #171

Closed VictorMatt01 closed 2 years ago

VictorMatt01 commented 2 years ago

So this question is about the algorithm when it uses 3 sentences as input: In the code on github the following "weight is used" when computing the final cos_sim score, but apperantly this score is 0. I can't really find an explanation for it in the paper and i was wondering how the cos_sim would change if the weight is changed (both in positive and negative direction)

Screenshot 2022-05-15 at 17 48 43
gaotianyu1350 commented 2 years ago

Here hard_negative_weight is the log of the hard negative weight (alpha in Table 7).

Golovneva commented 2 years ago

I guess I have the same question, what's the point of having summands that you multiply by 0? Like [0.0] * (cos_sim.size(-1) - z1_z3_cos.size(-1)) . Wont's this weights tensor always boil down to just [z3_weight]?

gaotianyu1350 commented 2 years ago

[0.0] * x will create an array of size x with all values as 0.