search

artidoro / frank

FRANK: Factuality Evaluation Benchmark
MIT License
52 stars 4 forks source link

What's the different between BertScore and BertScore Art? #8

Closed ariecattan closed 3 years ago

chenllliang commented 3 years ago

same question ! looking forward to author's reply !

artidoro commented 3 years ago

Hello! Thanks for your question.

BertScore compares two texts, usually the generated one and the reference. BertScore Art is run using the article as reference, while BertScore is using the target summary (provided as part of the summarization dataset) as reference. In theory, using the article should be better when measuring the factuality of a generated summary since the reference might not contain some important information.

Let me know if you need more information on this.

ariecattan commented 3 years ago

Thanks for the answer!

chenllliang commented 3 years ago

Thanks. I've submitted one result using SimCSE to do the evaluation.