Closed liuh236 closed 3 years ago
it seems that SentEval currently includes 17 downstream tasks but the code only use 14 tasks? but how to obtain ~76 results?
Hi,
Can you paste the error you mentioned here? I think it might be related to the error.
SentEval contains different types of tasks (probing, STS, and transfer tasks). In our paper we take the STS tasks for the main experiment, since it directly evaluates the quality of sentence embeddings.
when i use the training script(exactly the same as run_unsup_example.sh), the error is as follows:
Traceback (most recent call last):
File "train.py", line 584, in
Hi,
It seems that you are using multiple GPUs (because there is data_parallel
called)? Can you confirm that you are using a single GPU?
Hi, The logger info is fowllows:
You are right I am using multiple GPUs. But the script (run_unsup_example.sh) is a single-GPU (or CPU) example for the unsupervised version as ReadMe describes, doesn't it? what happened?
If you want to use single-GPU, you should use CUDA_VISIBLE_DEVICES
to control the GPU devices. If you use the single-GPU script on multiple GPUs, it will automatically use DataParallel and the batch will be sliced to different GPUs, leading to smaller negative sampling size. If you really want to use multiple GPUs, you should follow our guide/script for using distributed data parallel.
Thanks for your help! Finally, I get the right score ~76.
You did a nice job! I'm now closing the issue. Thanks again. 👍
You did a nice job! I'm now closing the issue. Thanks again. 👍
hi I have the same problem, how to change the unsupervised code
Hello, and thank you for this useful code! I tried to reproduce the unsupervisd BERT+SimCSE results, but failed. My environment setup is as follows:
pytorch=1.7.1 cudatoolkit=11.1 Single RTX 3090 The following script is the training script I used (exactly the same as run_unsup_example.sh).
python train.py \ --model_name_or_path bert-base-uncased \ --train_file data/wiki1m_for_simcse.txt \ --output_dir result/my-unsup-simcse-bert-base-uncased \ --num_train_epochs 1 \ --per_device_train_batch_size 64 \ --learning_rate 3e-5 \ --max_seq_length 32 \ --evaluation_strategy steps \ --metric_for_best_model stsb_spearman \ --load_best_model_at_end \ --eval_steps 125 \ --pooler_type cls \ --mlp_only_train \ --overwrite_output_dir \ --temp 0.05 \ --do_train \ --do_eval \ --fp16 \ "$@" However, there is a runtimeerror when training is finished. I obtained following evaluation results:
+-------+-------+-------+-------+-------+--------------+-----------------+-------+ | STS12 | STS13 | STS14 | STS15 | STS16 | STSBenchmark | SICKRelatedness | Avg. | +-------+-------+-------+-------+-------+--------------+-----------------+-------+ | 64.28 | 79.15 | 70.99 | 78.38 | 78.26 | 75.62 | 67.58 | 73.47 | +-------+-------+-------+-------+-------+--------------+-----------------+-------+
I think the gap (2.8 in average) is too large. Is it because of the error? How to obtain ~76 results in STS tasks?