search

artidoro / frank

FRANK: Factuality Evaluation Benchmark
MIT License
52 stars 4 forks source link

summary reference #2

Closed ZhangShiyue closed 3 years ago

ZhangShiyue commented 3 years ago

Hi,

Thanks for your awesome work!

I just wonder if you would consider include summary references in your benchmark data. Because probably people want to test their general summarization metrics on your benchmark too.

Shiyue

artidoro commented 3 years ago

Hey @ZhangShiyue, that's a great point! I have been thinking about article-based metrics, but I will add the reference summaries as well to allow testing reference-based metrics.

I'll let you know when that's done.

artidoro commented 3 years ago

@ZhangShiyue I just added reference summaries to the benchmark data file. Let me know if you have any other suggestions!

ZhangShiyue commented 3 years ago

Great! Thanks!