search

serenayj / PyrEval

Automated Pyramid Summarization Evaluation
GNU General Public License v3.0
11 stars 5 forks source link

One reference #11

Closed ZhangShiyue closed 3 years ago

ZhangShiyue commented 3 years ago

Hi,

I'm trying to use PyrEval for one-reference tasks like CNN/DM summarization. However, it will throw an error. When I repeat the reference twice, it works fine. But I don't think the numbers are valid because after repeating 3 or 4 times different numbers are given.

Could you please help me with this? Thanks!

Shiyue

serenayj commented 3 years ago

Hi Shiyue,

Thanks for your interest in PyrEval. The pyramid evaluation needs at least 3 reference summaries (4-5 ideally), because it evaluates the importance of contents based on their occurrence in distinct references (the "wise crowd" method). More details could be found in our CoNLL paper. With that said, PyrEval will not be able to evaluate if there is only one reference. Hope it helps!

ZhangShiyue commented 3 years ago

I see. Thank you so much!