Closed KawaiiNotHawaii closed 2 years ago
Hi,
You need to call evaluate.py separately to get the full numbers. The one in the training process only reports the DEV set performance.
So for sts tasks, the numbers in 'all' setting are reported, and for transfer tasks, the numbers in 'devacc' are reported? Would you please explain why not use 'acc' for transfer tasks?
Hi,
Are you talking about the evaluation.py or the in-training one? In in-training one it should be dev because it should only report the dev numbers. In evaluation.py you can choose to report dev or test numbers.
Hi, it seems that the evaluate function that is called after the training process doesn't report the performance for all sts tasks and transfer tasks? Did you actually use the evaluation.py in the 'test' setting to generate the reported results in the paper?