search

feralvam / easse

Easier Automatic Sentence Simplification Evaluation
GNU General Public License v3.0
157 stars 36 forks source link

result dismatch with paper #81

Closed monkdou0 closed 3 years ago

monkdou0 commented 3 years ago

the result of SARI and bleu to model is mismatch with the original paper WHY????? image

image

louismartin commented 3 years ago

Hello @monkdou0 Thanks for the issue. As mentioned in the the last section of the README, the EASSE implementation fixes bugs and inconsistencies with previous versions of SARI that were used in earlier works. The scores are not the same although similar. We highly recommend to use this fixed version and recompute scores using the system outputs provided in the library.

louismartin commented 3 years ago

If you use the latest version of EASSE, you will get these results:

$ easse evaluate -t turkcorpus_test -m sari,bleu -i easse/resources/data/system_outputs/turkcorpus/test/Hybrid
{'bleu': 50.707, 'sari': 31.497}

Papers which use the latest version of SARI include: Multilingual Unsupervised Sentence Simplification.

monkdou0 commented 3 years ago

my version is 580bba7e1378fc8289c663f864e0487188fe8067, and the result is ➜ easse evaluate -t turkcorpus_test -m sari,bleu -i easse/resources/data/system_outputs/turkcorpus/test/Hybrid {'bleu': 50.707, 'sari': 28.154}

monkdou0 commented 3 years ago

I have find what's wrong with me THx! @louismartin

louismartin commented 3 years ago

Hi @monkdou0 do you mind sharing what was wrong with us? It might help other people in the future!

monkdou0 commented 3 years ago

That is totally my own fault. i change the code of the implementation of SARI function several days ago, and compile it today. This repo is useful for the task of text simplification. Very good! thx

louismartin commented 3 years ago

Ok thanks, closing the issue then!