Is negative example shared within mini-batch?

princeton-nlp / SimCSE

[EMNLP 2021] SimCSE: Simple Contrastive Learning of Sentence Embeddings https://arxiv.org/abs/2104.08821

MIT License

3.41k stars 515 forks source link

Closed sh0416 closed 3 years ago

sh0416 commented 3 years ago

Hi, I wonder that the negative examples are shared when computing cosine similarity.

I think the broadcasting is not needed..

z1_z3_cos = cls.sim(z1.unsqueeze(1), z3.unsqueeze(0))

Also, could you give some commit id that reproduces the checkpoint you provided?

Thanks,

sh0416 commented 3 years ago

z1_z3_cos = cls.sim(z1, z3).unsqueeze(-1)

Could be better, I think.

gaotianyu1350 commented 3 years ago

Sorry I didn't get the question. Were you asking whether the negatives are shared within the batch? The answer is yes.

You should be able to reproduce our result by using the latest commit.