search

princeton-nlp / SimCSE

[EMNLP 2021] SimCSE: Simple Contrastive Learning of Sentence Embeddings https://arxiv.org/abs/2104.08821
MIT License
3.36k stars 507 forks source link

The dev Spearman is not correspond to the test Spearman for supervised model #177

Closed YJiangcm closed 2 years ago

YJiangcm commented 2 years ago

image Here shows that the eval_stsb_spearman and the eval_sickr_spearman of the current checkpoint are 0.8417 and 0.8118, respectively. Then I end the training and use evaluation.py to evaluate this checkpoint. However, the stsb_spearman and sickr_spearman on dev changed, as the below picture shows. This phenomenon only shows on supervised model but not unsupversied models. Would you mind explaining this? thanks a lot.

1653632285(1)

YJiangcm commented 2 years ago

I set "pooler_type" to cls both training and testing supervised simcse-bert-base-uncased.

gaotianyu1350 commented 2 years ago

Hi,

Before using the evaluate.py, you should convert the model to the hugging face format (in the readme).