search

Wendy-Xiao / redundancy_reduction_longdoc

This is the official code for the paper 'Systematically Exploring Redundancy Reduction inSummarizing Long Documents'.
MIT License
15 stars 3 forks source link

About the ORACLE on PubMed and ArXiv #2

Closed FortuneSeeker closed 3 years ago

FortuneSeeker commented 3 years ago

Hi, thx for your sharing the code. I have noticed that the ORACLE Rouge-L scores of the two datasets (PubMed & ArXiv) in this paper are totally different from that in your another article "Extractive Summarization of Long Documents by Combining Global and Local Context". Is this difference comes from the way the ORACLE is generated, can you please explain this? Thanks!

Wendy-Xiao commented 3 years ago

Hi, it is because we use different ways to compute ROUGE-L in the two papers (ROUGE-Lsum v.s. ROUGE-Lsent). You can find the ROUGE-L of the model Ext_summ_LG is also much higher than the original paper. Specifically, the difference is whether the sentences in both ground-truth and generated summary are aggregated with '\n' or ' '. In the former case, the ROUGE-L is on the sentence level, which will result in the ROUGE-L in this paper, and in the latter case, the ROUGE-L is on the summary level, resulting in the ROUGE-L reported in the original ext_summ_lg paper.

Finally, answer in one sentence, we use exactly the same oracle as in the paper 'Extractive Summarization of Long Documents by Combining Global and Local Context', but with a different way to compute ROUGE-L.

FortuneSeeker commented 3 years ago

oh, I got it! Thank you for the detailed explanation!