search

Unbabel / COMET

A Neural Framework for MT Evaluation
https://unbabel.github.io/COMET/html/index.html
Apache License 2.0
453 stars 72 forks source link

[QUESTION] Does COMET support Scoring multiple refs like scarebleu? #140

Closed dsj96 closed 1 year ago

dsj96 commented 1 year ago

❓ Questions and Help

Does COMET support Scoring multiple refs? Or do I need to calculate each ref separately and take the average as the final COMET score?

Looking forward to your reply. : )

ricardorei commented 1 year ago

Hi @dsj96,

As you mentioned, in order to utilize COMET with multiple references, all you have to do is average the scores obtained from the various references. We have previously investigated the influence of multiple references, and our findings indicate that when it comes to neural metrics like COMET, using multiple references is not as crucial as it is for lexical metrics such as BLEU. In fact, having a single high-quality reference appears to yield better results.

For more details, please refer to section 10.4 of this paper.

dsj96 commented 1 year ago

Thank you very much for your reply!