search

kavgan / ROUGE-2.0

ROUGE automatic summarization evaluation toolkit. Support for ROUGE-[N, L, S, SU], stemming and stopwords in different languages, unicode text evaluation, CSV output.
https://kavgan.github.io/ROUGE-2.0
Apache License 2.0
209 stars 37 forks source link

Long Running Time #12

Closed ShirleyHan6 closed 5 years ago

ShirleyHan6 commented 5 years ago

Hi, I am trying to get ROUGE-L of a dataset that contains around 10000 news articles and it has taken more than half an hour. Is that normal?

ShirleyHan6 commented 5 years ago

It has taken two hours and the process still has not finished.

kavgan commented 5 years ago

That is not normal, unless you have a lot of reference files. What is the screen output saying.

kavgan commented 5 years ago

Can you start by trying with a 100 articles and see if there is a problem?

ShirleyHan6 commented 5 years ago

I ran rouge-1 successfully on around 200 articles. But it seems to take very long for ROUGE-L with around 10000

The screen output was like this for two hours: 2019-02-14 22:57:06 INFO ROUGECalculator:203 - Working on task1 ngram=L

ShirleyHan6 commented 5 years ago

Is there any way to speed it up:)

kavgan commented 5 years ago

I see, if you can send me a dump of your reference, system, etc. I can take a look at why its taking so long? But it won't be immediate though, it will be in the next couple of weeks. :-)

ailsamm commented 5 years ago

@kavgan Was there a solution to this? I'm having exactly the same problem. ROUGE-1 and ROUGE-2 are taking the expected amount of time, but ROUGE-L has been going for about 4 hours now.

kavgan commented 5 years ago

How many tasks are you evaluating? For a few hundred files it shouldn't take that long. How long did ROUGE 1 & 2 take? If you email a dump of files I can run a test to see what's going on.