search

neulab / REALSumm

REALSumm: Re-evaluating Evaluation in Text Summarization
Creative Commons Attribution 4.0 International
71 stars 4 forks source link

BART included twice #3

Open danieldeutsch opened 3 years ago

danieldeutsch commented 3 years ago

Hi,

Thanks for this exciting work. Both the dataset and analysis will be very useful!

It looks like BART is listed as both extractive (#8) and abstractive (#13). I believe the files are also identical. It also appears twice in the pickle files with the scores, both under the system name "bart_out".

Is this a mistake or is there a misunderstanding on my part?

Thanks!

ZhangShiyue commented 3 years ago

looks like they have different summaries (one is abstractive, the other is extractive) if you check the two bart files in human_annotations https://github.com/neulab/REALSumm/blob/master/human_annotations/SCUs_and_annotations/SCU_rating_results_abs_systems/bart_out_Batch_crowd_3.csv and https://github.com/neulab/REALSumm/blob/master/human_annotations/SCUs_and_annotations/SCU_rating_results_ext_systems/bart_out.csv

However, in the pickle files, they are indeed identical. Looks like they both are the extractive summaries from https://github.com/neulab/REALSumm/blob/master/human_annotations/SCUs_and_annotations/SCU_rating_results_ext_systems/bart_out.csv.

So I guess they might make a mistake when compiling pickle files?