search

google-research-datasets / xsum_hallucination_annotations

Faithfulness and factuality annotations of XSum summaries from our paper "On Faithfulness and Factuality in Abstractive Summarization" (https://www.aclweb.org/anthology/2020.acl-main.173.pdf).
80 stars 6 forks source link

some questions about the annotations #4

Closed xieyxclack closed 3 years ago

xieyxclack commented 3 years ago

Many thanks for providing such valuable annotation data!

I find some items in the file xsum_hallucination_annotations/eval_scores_xsum_summaries.csv do not appear in the related file xsum_hallucination_annotations/xsum_hallucination_annotations.
for example, ``bert_withckpt_39263182 bert_withckpt_37972178 bert_nockpt_25840979 ptgen_40630842 ptgen_40899859 bert_withckpt_13193011 ptgen_39964844 bert_withckpt_36128472 tconvs2s_26065545 tconvs2s_33960105 bert_withckpt_37900100 ptgen_37972178''

And I notice that the faithful and factual scores of these items are 1.0 (from file xsum_hallucination_annotations/eval_scores_xsum_summaries.csv) And I want to get the generated summary (from the file xsum_hallucination_annotations/xsum_hallucination_annotations) for more analysis, but fail.

Hope to hear from you :) Thanks

shashiongithub commented 3 years ago

Hi there, I missed to reply to this.

First of all, please use the following mapping:

ptgen: PtGen tconvs2s: TConvS2S bert_nockpt: TranS2S bert_withckpt : BERTS2S

For bert_withckpt_39263182, for example, I find them

39263182,BERTS2S,gateshead remain unbeaten in the national league after being held to a draw by guiseley.,NULL,NULL,-1,-1,wid_0