search

neulab / BARTScore

BARTScore: Evaluating Generated Text as Text Generation
Apache License 2.0
318 stars 37 forks source link

Question about unreferenced usage #37

Closed susisheng closed 1 year ago

susisheng commented 1 year ago

In paper CTRLEval: An Unsupervised Reference-Free Metric for Evaluating Controlled Text Generation, BARTScore is used as a baseline in an unreferenced way but I can't find relative notes about that. Could you pls show how to perform the reference-free evaluation? Thank you very much!

yyy-Apple commented 1 year ago

Hi, for using BARTScore in a reference-free way, you just need to use BARTScore to calculate the probability of generating the actual target given the source. For example, like below.

bart_scorer.score(['This is the document'], ['This is the summary.'], batch_size=4)
susisheng commented 1 year ago

Got it! Thank you very much!