No score in STS15 - Githubissues

facebookresearch / SentEval

A python tool for evaluating the quality of sentence embeddings.

Other

2.09k stars 309 forks source link

No score in STS15 #85

Closed sh0416 closed 3 years ago

sh0416 commented 3 years ago

Hi, I am curious about the dataset STS15.

In STS15, some rows doesn't have score and the code ignores these examples.

Is there any reason that the lines exists?
I think the dataset statistics is inappropriate if we ignored. Could you correct it?

noooob-coder commented 3 years ago

I also encountered this problem recently. Have you solved it？

sh0416 commented 3 years ago

@noooob-coder I followed the process that they took.

Just remove the rows that are not completed.

In my case, I think the reason that the statistic is unfair when we compare the evaluated result with the statistics that some papers reported, which is not mentioned about these details.

So, I just evaluate other methods on the same framework.