saharghannay
commented
5 years ago Hi, I want to correct something : I download the data with get_transfer_data.bash from SenEval.
Open saharghannay opened 5 years ago
saharghannay
commented
5 years ago Hi, I want to correct something : I download the data with get_transfer_data.bash from SenEval.
Hi,
I evaluate the InferSent 1 model on STS transfer tasks (STS12-16 and STSBenchmark) using the SentEval tool. However I could not reproduce the same results.
2019-04-28 17:35:35,511 : Transfer task : STS12 Found 8104(/8122) words with w2v vectors Vocab size : 8104 2019-04-28 17:35:37,836 : ALL (weighted average) : Pearson = 0.5893, Spearman = 0.5896 2019-04-28 17:35:37,836 : ALL (average) : Pearson = 0.5792, Spearman = 0.5798
2019-04-28 17:35:37,836 : Transfer task : STS13 (-SMT) Found 5106(/5116) words with w2v vectors Vocab size : 5106 2019-04-28 17:35:38,933 : ALL (weighted average) : Pearson = 0.6399, Spearman = 0.6371 2019-04-28 17:35:38,933 : ALL (average) : Pearson = 0.5553, Spearman = 0.5541
2019-04-28 17:35:38,933 : Transfer task : STS14 Found 9046(/9060) words with w2v vectors Vocab size : 9046 2019-04-28 17:35:41,391 : ALL (weighted average) : Pearson = 0.6545, Spearman = 0.6377 2019-04-28 17:35:41,391 : ALL (average) : Pearson = 0.6411, Spearman = 0.6258
2019-04-28 17:35:41,391 : * Transfer task : STS15 Found 7313(/7335) words with w2v vectors Vocab size : 7313 2019-04-28 17:35:43,532 : ALL (weighted average) : Pearson = 0.6879, Spearman = 0.6988 2019-04-28 17:35:43,533 : ALL (average) : Pearson = 0.6600, Spearman = 0.6731
2019-04-28 17:35:43,533 : Transfer task : STS16 Found 3930(/3971) words with w2v vectors Vocab size : 3930 2019-04-28 17:35:44,572 : ALL (weighted average) : Pearson = 0.6287, Spearman = 0.6465 2019-04-28 17:35:44,572 : ALL (average) : Pearson = 0.6321, Spearman = 0.6494
Transfer task : STSBenchmark Found 15718(/15755) words with w2v vectors Vocab size : 15718 2019-04-28 17:36:19,249 : Test : Pearson 0.68976421922 Spearman 0.683016070529
PS: I used /dataset/get_data.bash to download the data,
for word embeddings I was using CBOW trained on wiki data.
Is there something wrong with the default setting?
Thanks for your help!