danieldeutsch / sacrerouge

SacreROUGE is a library dedicated to the use and development of text generation evaluation metrics with an emphasis on summarization.
Apache License 2.0
134 stars 11 forks source link

Add implementation of modified Pyramid Score #20

Closed danieldeutsch closed 4 years ago

danieldeutsch commented 4 years ago

Given a Pyramid and a PyramidAnnotation, calculate the modified pyramid score. It will almost definitely be different than what is reported in DUC and TAC because the pyramid files often had errors, so our pyramids are very slightly different than what was used in the original calculations.

We should also include experiments that show the correlation to the original score as well as the responsiveness.