I evaluate the InferSent 1 model on STS transfer tasks (STS12-16 and STSBenchmark) using the SentEval tool.
However I could not reproduce the same results.
2019-04-28 17:35:35,511 : Transfer task : STS12
Found 8104(/8122) words with w2v vectors
Vocab size : 8104
2019-04-28 17:35:37,836 : ALL (weighted average) : Pearson = 0.5893, Spearman = 0.5896
2019-04-28 17:35:37,836 : ALL (average) : Pearson = 0.5792, Spearman = 0.5798
2019-04-28 17:35:37,836 : Transfer task : STS13 (-SMT)
Found 5106(/5116) words with w2v vectors
Vocab size : 5106
2019-04-28 17:35:38,933 : ALL (weighted average) : Pearson = 0.6399, Spearman = 0.6371
2019-04-28 17:35:38,933 : ALL (average) : Pearson = 0.5553, Spearman = 0.5541
2019-04-28 17:35:38,933 : Transfer task : STS14
Found 9046(/9060) words with w2v vectors
Vocab size : 9046
2019-04-28 17:35:41,391 : ALL (weighted average) : Pearson = 0.6545, Spearman = 0.6377
2019-04-28 17:35:41,391 : ALL (average) : Pearson = 0.6411, Spearman = 0.6258
2019-04-28 17:35:41,391 : * Transfer task : STS15
Found 7313(/7335) words with w2v vectors
Vocab size : 7313
2019-04-28 17:35:43,532 : ALL (weighted average) : Pearson = 0.6879, Spearman = 0.6988
2019-04-28 17:35:43,533 : ALL (average) : Pearson = 0.6600, Spearman = 0.6731
2019-04-28 17:35:43,533 : Transfer task : STS16
Found 3930(/3971) words with w2v vectors
Vocab size : 3930
2019-04-28 17:35:44,572 : ALL (weighted average) : Pearson = 0.6287, Spearman = 0.6465
2019-04-28 17:35:44,572 : ALL (average) : Pearson = 0.6321, Spearman = 0.6494
Transfer task : STSBenchmark
Found 15718(/15755) words with w2v vectors
Vocab size : 15718
2019-04-28 17:36:19,249 : Test : Pearson 0.68976421922 Spearman 0.683016070529
PS: I used /dataset/get_data.bash to download the data,
for word embeddings I was using CBOW trained on wiki data.
Is there something wrong with the default setting?
Hi,
I evaluate the InferSent 1 model on STS transfer tasks (STS12-16 and STSBenchmark) using the SentEval tool. However I could not reproduce the same results.
2019-04-28 17:35:35,511 : Transfer task : STS12 Found 8104(/8122) words with w2v vectors Vocab size : 8104 2019-04-28 17:35:37,836 : ALL (weighted average) : Pearson = 0.5893, Spearman = 0.5896 2019-04-28 17:35:37,836 : ALL (average) : Pearson = 0.5792, Spearman = 0.5798
2019-04-28 17:35:37,836 : Transfer task : STS13 (-SMT) Found 5106(/5116) words with w2v vectors Vocab size : 5106 2019-04-28 17:35:38,933 : ALL (weighted average) : Pearson = 0.6399, Spearman = 0.6371 2019-04-28 17:35:38,933 : ALL (average) : Pearson = 0.5553, Spearman = 0.5541
2019-04-28 17:35:38,933 : Transfer task : STS14 Found 9046(/9060) words with w2v vectors Vocab size : 9046 2019-04-28 17:35:41,391 : ALL (weighted average) : Pearson = 0.6545, Spearman = 0.6377 2019-04-28 17:35:41,391 : ALL (average) : Pearson = 0.6411, Spearman = 0.6258
2019-04-28 17:35:41,391 : * Transfer task : STS15 Found 7313(/7335) words with w2v vectors Vocab size : 7313 2019-04-28 17:35:43,532 : ALL (weighted average) : Pearson = 0.6879, Spearman = 0.6988 2019-04-28 17:35:43,533 : ALL (average) : Pearson = 0.6600, Spearman = 0.6731
2019-04-28 17:35:43,533 : Transfer task : STS16 Found 3930(/3971) words with w2v vectors Vocab size : 3930 2019-04-28 17:35:44,572 : ALL (weighted average) : Pearson = 0.6287, Spearman = 0.6465 2019-04-28 17:35:44,572 : ALL (average) : Pearson = 0.6321, Spearman = 0.6494
Transfer task : STSBenchmark Found 15718(/15755) words with w2v vectors Vocab size : 15718 2019-04-28 17:36:19,249 : Test : Pearson 0.68976421922 Spearman 0.683016070529
PS: I used /dataset/get_data.bash to download the data,
for word embeddings I was using CBOW trained on wiki data.
Is there something wrong with the default setting?
Thanks for your help!