Question about avg_first_last pooling

princeton-nlp / SimCSE

[EMNLP 2021] SimCSE: Simple Contrastive Learning of Sentence Embeddings https://arxiv.org/abs/2104.08821

MIT License

3.36k stars 507 forks source link

Question about avg_first_last pooling #133

Closed qishibo closed 2 years ago

qishibo commented 2 years ago

https://github.com/princeton-nlp/SimCSE/blob/main/simcse/models.py#L72-L76

avg_first_lastshould be the mean of the first and last layers，but hidden_states[0] is the input embedding, hidden_states[1]is the first layer output, should't it be first_hidden = hidden_states[1]??

gaotianyu1350 commented 2 years ago

Hi,

Thanks for pointing this out! You are right that 0-index is the embedding layer and it has been fixed now. Though it does not affect our results in the paper since they all use CLS or last layer average.