Closed ypuzikov closed 2 years ago
After reading the closed issues, I tried converting the weights to the HF checkpoint. Got slightly better results, but by not much:
------ test ------
+-------+-------+-------+-------+-------+--------------+-----------------+-------+
| STS12 | STS13 | STS14 | STS15 | STS16 | STSBenchmark | SICKRelatedness | Avg. |
+-------+-------+-------+-------+-------+--------------+-----------------+-------+
| 50.92 | 55.15 | 57.24 | 67.02 | 60.67 | 58.67 | 55.69 | 57.91 |
+-------+-------+-------+-------+-------+--------------+-----------------+-------+
I saw the issue #38 which suggests disabling the --fp16
option. Will retrain the model and report back later.
Reporting back:
run_run_unsup_example.sh
script, but without the --fp16
flagsimcse_to_huggingface.py
script------ test ------
+-------+-------+-------+-------+-------+--------------+-----------------+-------+
| STS12 | STS13 | STS14 | STS15 | STS16 | STSBenchmark | SICKRelatedness | Avg. |
+-------+-------+-------+-------+-------+--------------+-----------------+-------+
| 50.06 | 55.05 | 57.67 | 67.48 | 60.69 | 57.63 | 56.02 | 57.80 |
+-------+-------+-------+-------+-------+--------------+-----------------+-------+
Results with the --pooler cls_before_pooler
flag:
------ test ------
+-------+-------+-------+-------+-------+--------------+-----------------+-------+
| STS12 | STS13 | STS14 | STS15 | STS16 | STSBenchmark | SICKRelatedness | Avg. |
+-------+-------+-------+-------+-------+--------------+-----------------+-------+
| 49.49 | 55.75 | 58.12 | 66.50 | 60.36 | 57.28 | 56.59 | 57.73 |
+-------+-------+-------+-------+-------+--------------+-----------------+-------+
Environment:
Hi, you should use the cls_before_pooler
for evaluation (and that is where most not-reproducible cases come from). But since you also tried that, I suggest you to evaluate our pre-trained checkpoints first (to make sure nothing goes wrong with the evaluation). You can also check the log for training, where validation results are printed.
Hi!
Running python evaluation.py --model_name_or_path princeton-nlp/sup-simcse-bert-base-uncased --pooler cls --task_set sts --mode test
gives me the following results:
------ test ------
+-------+-------+-------+-------+-------+--------------+-----------------+-------+
| STS12 | STS13 | STS14 | STS15 | STS16 | STSBenchmark | SICKRelatedness | Avg. |
+-------+-------+-------+-------+-------+--------------+-----------------+-------+
| 75.30 | 84.67 | 80.19 | 85.40 | 80.82 | 84.26 | 80.39 | 81.58 |
+-------+-------+-------+-------+-------+--------------+-----------------+-------+
That matches the expected result :)
By the way, running the same command, but w/ the cls_before_pooler
option gives:
------ test ------
+-------+-------+-------+-------+-------+--------------+-----------------+-------+
| STS12 | STS13 | STS14 | STS15 | STS16 | STSBenchmark | SICKRelatedness | Avg. |
+-------+-------+-------+-------+-------+--------------+-----------------+-------+
| 77.75 | 81.43 | 79.53 | 85.59 | 81.98 | 83.68 | 79.52 | 81.35 |
+-------+-------+-------+-------+-------+--------------+-----------------+-------+
In any case, it seems like the results from your pretrained models are computed as expected, so eval code works. Any hints on where the problem w/ the train code might be?
Can you check the validation results for debugging?
Even compared to the test scores in Table 5, the validation results on dev set are fairly low -- here are the highest scores I got:
{'eval_stsb_spearman': 0.6589651676962802, 'eval_sickr_spearman': 0.595184228171256, 'eval_avg_sts': 0.6270746979337681, 'epoch': 0.62}
The validation result doesn't seem right, so there is indeed something going wrong during the training. Since it's hard to debug for me, can you try to re-clone the github repo, redownload all the data and train again (just to make sure code/data are not corrupted)?
Did that and also asked another person to do that -- same result “¯_(ツ)_/¯“ Can you confirm that a fresh clone results in smth different for you?
I ran into exactly the same question as @ypuzikov, please have a look at your latest repo, thanks!
I ran into exactly the same question as @ypuzikov, please have a look at your latest repo, thanks!
I just delete the previous contents in result folder and rerun the script run_unsup_example.sh
. This model gives me the following results:
------ test ------
+-------+-------+-------+-------+-------+--------------+-----------------+-------+
| STS12 | STS13 | STS14 | STS15 | STS16 | STSBenchmark | SICKRelatedness | Avg. |
+-------+-------+-------+-------+-------+--------------+-----------------+-------+
| 68.99 | 82.87 | 73.33 | 79.40 | 77.32 | 75.63 | 69.58 | 75.30 |
+-------+-------+-------+-------+-------+--------------+-----------------+-------+
It seems like an acceptable results compare with the results reported in the paper though I still don't know the cause of the problem I came across yesterday...
This looks reasonable to me. @ypuzikov Can you try @SincereAlex's suggestion, deleting all the result and running it again?
Yeah, will do that in the next couple of days and come back. Although, I have to say, the solution looks more like magic to me -- why would wiping the directory improve the results?
Hey, everyone! It took a bit longer than a couple of days :) Anyway, the hack did not work -- here are my results after wiping out the results folder and re-training the model:
------ test ------
+-------+-------+-------+-------+-------+--------------+-----------------+-------+
| STS12 | STS13 | STS14 | STS15 | STS16 | STSBenchmark | SICKRelatedness | Avg. |
+-------+-------+-------+-------+-------+--------------+-----------------+-------+
| 49.49 | 55.75 | 58.12 | 66.50 | 60.36 | 57.28 | 56.59 | 57.73 |
+-------+-------+-------+-------+-------+--------------+-----------------+-------+
Hi, I just cloned the github repo and rerun the experiment, and I can still reproduce the result. I'm not sure what went wrong. Could you please try it out on some other environments/machines?
Unfortunately, I also cannot reproduce the results reported. Using the current GitHub repo. I ran the script:
sh run_unsup_example.sh
I evaluated on different machines:
python evaluation.py --model_name_or_path result/my-unsup-simcse-bert-base-uncased --pooler cls --task_set sts --mode test
and I get this result:
AWS Deep Learning AMI (Ubuntu 18.04) Version 47.0 V100 (p3.2xlage) Transformers 4.2.1 PyTorch 1.9.0 CUDA Version: 11.0
------ test ------
+-------+-------+-------+-------+-------+--------------+-----------------+-------+
| STS12 | STS13 | STS14 | STS15 | STS16 | STSBenchmark | SICKRelatedness | Avg. |
+-------+-------+-------+-------+-------+--------------+-----------------+-------+
| 69.98 | 82.50 | 73.83 | 81.88 | 79.41 | 78.23 | 70.58 | 76.63 |
+-------+-------+-------+-------+-------+--------------+-----------------+-------+
+-------+-------+-------+-------+-------+-------+-------+-------+
| MR | CR | SUBJ | MPQA | SST2 | TREC | MRPC | Avg. |
+-------+-------+-------+-------+-------+-------+-------+-------+
| 82.88 | 89.20 | 94.81 | 89.67 | 87.31 | 88.40 | 73.51 | 86.54 |
+-------+-------+-------+-------+-------+-------+-------+-------+
And on another machine:
AWS Deep Learning AMI (Ubuntu 18.04) Version 50.0 V100 (p3.2xlage) Transformers 4.2.1 PyTorch 1.7.1 CUDA Version: 11.0
------ test ------
+-------+-------+-------+-------+-------+--------------+-----------------+-------+
| STS12 | STS13 | STS14 | STS15 | STS16 | STSBenchmark | SICKRelatedness | Avg. |
+-------+-------+-------+-------+-------+--------------+-----------------+-------+
| 68.05 | 80.38 | 72.62 | 78.96 | 76.90 | 75.11 | 69.37 | 74.48 |
+-------+-------+-------+-------+-------+--------------+-----------------+-------+
+-------+-------+-------+-------+-------+-------+-------+-------+
| MR | CR | SUBJ | MPQA | SST2 | TREC | MRPC | Avg. |
+-------+-------+-------+-------+-------+-------+-------+-------+
| 80.74 | 85.75 | 93.96 | 88.60 | 84.57 | 86.20 | 73.51 | 84.76 |
+-------+-------+-------+-------+-------+-------+-------+-------+
AWS Deep Learning AMI (Ubuntu 18.04) Version 47.0 V100 (p3.2xlage) Transformers 4.10.0 dev PyTorch 1.7.1 CUDA Version: 11.0
------ test ------
+-------+-------+-------+-------+-------+--------------+-----------------+-------+
| STS12 | STS13 | STS14 | STS15 | STS16 | STSBenchmark | SICKRelatedness | Avg. |
+-------+-------+-------+-------+-------+--------------+-----------------+-------+
| 70.06 | 80.64 | 73.70 | 80.86 | 76.93 | 75.60 | 71.20 | 75.57 |
+-------+-------+-------+-------+-------+--------------+-----------------+-------+
+-------+-------+-------+-------+-------+-------+-------+-------+
| MR | CR | SUBJ | MPQA | SST2 | TREC | MRPC | Avg. |
+-------+-------+-------+-------+-------+-------+-------+-------+
| 80.74 | 86.68 | 94.27 | 88.97 | 86.16 | 85.40 | 73.51 | 85.10 |
+-------+-------+-------+-------+-------+-------+-------+-------+
Hi,
The first results are even higher than the reported numbers in the paper right? What is the issue here?
Hi, folks! Thank you very much for the hard work (^^) I have a question on how to reproduce the results -- not that I am aiming to spot the differences, just making sure that I am running the code correctly.
I use the
run_run_unsup_example.sh
script to train the unsupervised SimCSE. At the end of the training procedure, I run evaluation as follows:time CUDA_VISIBLE_DEVICES=0 python evaluation.py --model_name_or_path result/my-unsup-simcse-bert-base-uncased --pooler cls --task_set sts --mode test
. The results table I am getting is:I believe the table i should be comparing to is Table 5 from the paper, the relevant row shows:
Which is far better than what i get. Can you maybe help me understand if I am doing smth wrong? I follow the main README.md file, the content of the
run_run_unsup_example.sh
script is: