Closed CSerxy closed 2 years ago
Thanks for your interest in our work!
For you question, the “all” setting is relative to the “wmean” setting. Our code is a modified version of SentEval so the key we used here is different. If you use the original SentEval, it should be “all”. And your understanding about the SentEval keys are correct.
Hi,
I found in your evaluation code (i.e., evaluation.py) you computed the score by using the below line (i.e., Line 156)
"scores.append("%.2f" % (results[task]['dev']['spearman'][0] * 100))"
, which means it reports the spearman of dev on the STS-B and SICK-R datasets.
However, in Table 5 of your paper, you said the results are Spearman’s correlation in “all” setting. So I wonder which shall I refer to?
There are "ALL", "ALL (weighted average)", "ALL (average)" in the returned results from SentEval. I suppose the "all" setting you mentioned should be the "ALL" instead of "ALL (weighted average)" and "ALL (average)." Is it correct?
Many thanks!