search

salesforce / ctrl-sum

Resources for the "CTRLsum: Towards Generic Controllable Text Summarization" paper
https://arxiv.org/abs/2012.04281
BSD 3-Clause "New" or "Revised" License
146 stars 24 forks source link

Oracle entity in Table 2 VS. Oracle keywords in Table 7 #10

Open lifelongeek opened 3 years ago

lifelongeek commented 3 years ago

I am trying to reproduce ROUGE on CNNDM with 'oracle keyword in Table 7'. 'oracle entity setting in Table 2' sounds similar to 'oracle keyword in Table 7', however, ROUGE score is very different. Could you explain how these settings are different?

image

jxhe commented 3 years ago

Hi,

"Oracle entity" in Table 2 uses only the entity words in the groud-truth target, while "oracle keywords" contains non-entity words as well, as described in the paper

lifelongeek commented 3 years ago

Thanks for the clarification. I have some follow-up questions.

Does example_dataset/test.oraclewordns imply "oracle keywords"? Does "longest sub-sequences" used for training automatic keyword extractor imply "oracle keywords"? image

jxhe commented 3 years ago
  1. Yes, example_dataset/test.oraclewordns imply "oracle keywords"
  2. The keywords used for training automatic keyword extractor are "oracle keywords", yet strictly speaking "oracle keywords" are not exactly "longest sub-sequences" -- as described in your screenshot, "we remove duplicate words and stop words and keep the remaining tokens as keywords"
Wendy-Xiao commented 2 years ago

Hi,

I have a quick follow-up question on this point. For 'oracle entities', which NER tool did you used for extacting oracle entities from the reference summary?

Thanks a lot!!

jxhe commented 2 years ago

Hi, we use stanza for NER, you may refer to some examples here: https://github.com/salesforce/ctrl-sum/blob/6468beaaceebf463b492992fffef0e4f693a3281/scripts/preprocess.py#L890