Closed ShangQingTu closed 3 years ago
yixinL7
commented
3 years ago Hi ShangQing, thanks for your interest in our work!
It is normal that some of the generated candidates are very similar, but most of them should be different enough.
To reproduce the results, you should use the default setting with num_return_sequences=16, num_beam_groups=16, diversity_penalty=1.0, num_beams=16. Did you try this already?
Let me know if you have more questions.
ShangQingTu
commented
3 years ago Hi, yixin, thanks for your reply!
I use the article from your 0.json for example:
from transformers import BartTokenizer, BartForConditionalGeneration
bart_path = 'facebook/bart-large-cnn'
model = BartForConditionalGeneration.from_pretrained(bart_path)
tokenizer = BartTokenizer.from_pretrained(bart_path)
article = ["club tijuana star juan arango conjured memories luis suarez in his team 's 4-3 defeat by monterrey in the mexican league - but it was not through prodigious scoring .", "the venezuelan icon arango sank his teeth into the shoulder of jesus zavela as his temper flared in the defeat .", ".", "he was not booked by the referee but could face a heavy retrospective ban .", ".", "juan arango -lrb- left -rrb- bites the shoulder of opponent jesus zavela in a moment of madness .", "zavala holds his shoulder after being bitten by arango , in the game zavala 's side won 4-3 in mexico .", "zavala shows the referee the mark on his shoulder after being bittern by arango .", "arango -lrb- right -rrb- earlier scored a magnificent free kick to bring his club tijuana team level against monterrey .", "arango had earlier curled in a magnificent free kick for his team to bring them level after falling 2-0 down early on in the encounter .", "but the 34-year-old overshadowed his goal with the bite as television cameras picked up the moment of madness .", "arango spent 10 years playing in europe , spending five seasons each at real mallorca in spain and borussia monchengladbach in germany .", "he has made 121 appearances for venezuela .", "."]
ARTICLE_TO_SUMMARIZE = ' '.join(article)
inputs = tokenizer([ARTICLE_TO_SUMMARIZE], max_length=1024,truncation=True, return_tensors='pt')
print(inputs['input_ids'].shape)
# Generate Summary
summary_ids = model.generate(inputs['input_ids'],
num_return_sequences=16, num_beam_groups=16, diversity_penalty=1.0, num_beams=16)
abstract_list = [tokenizer.decode(g, skip_special_tokens=True, clean_up_tokenization_spaces=False) for g in summary_ids]
for abs in abstract_list:
print(abs)output is:
torch.Size([1, 564])
club tijuana star juan arango conjured memories luis suarez in his team 's 4-3 defeat by monterrey. The venezuelan icon sank his teeth into the shoulder of jesus zavela. He was not booked by the referee but could face a heavy retrospective ban.
club tijuana star juan arango conjured memories luis suarez in his team 's 4-3 defeat by monterrey. The venezuelan icon sank his teeth into the shoulder of jesus zavela. He was notbooked by the referee but could face a heavy retrospective ban.
club tijuana star juan arango conjured memories luis suarez in his team 's 4-3 defeat by monterrey. The venezuelan icon sank his teeth into the shoulder of jesus zavela. He was not booked by the referee, but could face a heavy retrospective ban.
club tijuana star juan arango conjured memories luis suarez in his team 's 4-3 defeat by monterrey. The venezuelan icon sank his teeth into the shoulder of jesus zavela. He was not booked by the referee but could face a heavy retrospective ban.
club tijuana star juan arango conjured memories luis suarez in his team 's 4-3 defeat by monterrey. The venezuelan icon sank his teeth into the shoulder of jesus zavela. He was notbooked by the referee, but could face a heavy retrospective ban.
club tijuana star juan arango conjured memories luis suarez in his team 's 4-3 defeat by monterrey. The venezuelan icon sank his teeth into the shoulder of jesus zavela. He was notbooked by the referee but could face a heavy retrospective ban.
club tijuana star juan arango conjured memories luis suarez in his team 's 4-3 defeat by monterrey. The venezuelan icon sank his teeth into the shoulder of jesus zavela as his temper flared.
club tijuana star juan arango conjured memories luis suarez in his team 's 4-3 defeat by monterrey. The venezuelan icon sank his teeth into the shoulder of jesus zavela.
club tijuana star juan arango conjured memories luis suarez in his team 's 4-3 defeat by monterrey. The venezuelan icon sank his teeth into the shoulder of jesus zavela. His temper flared in the defeat.
club tijuana star juan arango conjured memories luis suarez in his team 's 4-3 defeat by monterrey. The venezuelan icon sank his teeth into the shoulder of jesus zavela.
club tijuana star juan arango conjured memories luis suarez in his team 's 4-3 defeat by monterrey. The venezuelan icon sank his teeth into the shoulder of jesus zavela.
club tijuana star juan arango conjured memories luis suarez in his team 's 4-3 defeat by monterrey. The venezuelan icon sank his teeth into the shoulder of jesus zavela.
club tijuana star juan arango conjured memories luis suarez in his team 's 4-3 defeat by monterrey. The venezuelan icon sank his teeth into the shoulder of jesus zavela.
club tijuana star juan arango conjured memories luis suarez in his team 's 4-3 defeat by monterrey. The venezuelan icon sank his teeth into the shoulder of jesus zavela.
club tijuana star juan arango conjured memories luis suarez in his team's 4-3 defeat by monterrey. The venezuelan icon sank his teeth into the shoulder of jesus zavela.
club tijuana star juan arango conjured memories luis suarez in his team 's 4-3 defeat by monterrey. The venezuelan icon sank his teeth into the shoulder of jesus zavela.It looks like that the first sentence in every generated summary is the same. I don't know why. Maybe because I use the 'facebook/bart-large-cnn' which is fine-tuned so well that it overfits. Did you use 'facebook/bart-large-cnn'?
Thanks!
yixinL7
commented
3 years ago Thanks! First, yes I used 'facebook/bart-large-cnn'.
I've just tried with your setting. I think this issue is related to the version of the Transformers library. When I used version 3.1.0 which is used to train our evaluation model, I got similar results as you did. Using Transformers 4.1.1 and PyTorch 1.4 should get the result you want, which is what we used in our experiments.
Some more comments: When you use BART to generate the candidates, you should pass the model with cased, untokenized source documents. This discussion may be helpful https://github.com/pytorch/fairseq/issues/1401
This code snippet may also be helpful (what we used to generate the candidates):
max_length = 140
min_length = 55
dct = tokenizer.batch_encode_plus(slines, max_length=1024, return_tensors="pt", pad_to_max_length=True, truncation=True)
summaries = model.generate(
input_ids=dct["input_ids"].to(device),
attention_mask=dct["attention_mask"].to(device),
num_return_sequences=16, num_beam_groups=16, diversity_penalty=1.0, num_beams=16,
max_length=max_length + 2, # +2 from original because we start at step=1 and stop before max_length
min_length=min_length + 1, # +1 from original because we start at step=1
no_repeat_ngram_size=3,
length_penalty=2.0,
early_stopping=True,
)
dec = [tokenizer.decode(g, skip_special_tokens=True, clean_up_tokenization_spaces=False) for g in summaries]Thanks a lot! I just tried your setting and changed the transformers version from 3.1.0 to 4.1.1.
from transformers import BartTokenizer, BartForConditionalGeneration
bart_path = 'facebook/bart-large-cnn'
device = 'cuda'
model = BartForConditionalGeneration.from_pretrained(bart_path)
model.to(device)
tokenizer = BartTokenizer.from_pretrained(bart_path)
article = ["club tijuana star juan arango conjured memories luis suarez in his team 's 4-3 defeat by monterrey in the mexican league - but it was not through prodigious scoring .", "the venezuelan icon arango sank his teeth into the shoulder of jesus zavela as his temper flared in the defeat .", ".", "he was not booked by the referee but could face a heavy retrospective ban .", ".", "juan arango -lrb- left -rrb- bites the shoulder of opponent jesus zavela in a moment of madness .", "zavala holds his shoulder after being bitten by arango , in the game zavala 's side won 4-3 in mexico .", "zavala shows the referee the mark on his shoulder after being bittern by arango .", "arango -lrb- right -rrb- earlier scored a magnificent free kick to bring his club tijuana team level against monterrey .", "arango had earlier curled in a magnificent free kick for his team to bring them level after falling 2-0 down early on in the encounter .", "but the 34-year-old overshadowed his goal with the bite as television cameras picked up the moment of madness .", "arango spent 10 years playing in europe , spending five seasons each at real mallorca in spain and borussia monchengladbach in germany .", "he has made 121 appearances for venezuela .", "."]
slines = [' '.join(article)]
max_length = 140
min_length = 55
dct = tokenizer.batch_encode_plus(slines, max_length=1024, return_tensors="pt", pad_to_max_length=True, truncation=True)
print(dct["input_ids"].shape)
summaries = model.generate(
input_ids=dct["input_ids"].to(device),
attention_mask=dct["attention_mask"].to(device),
num_return_sequences=16, num_beam_groups=16, diversity_penalty=1.0, num_beams=16,
max_length=max_length + 2, # +2 from original because we start at step=1 and stop before max_length
min_length=min_length + 1, # +1 from original because we start at step=1
no_repeat_ngram_size=3,
length_penalty=2.0,
early_stopping=True,
)
abstract_list = [tokenizer.decode(g, skip_special_tokens=True, clean_up_tokenization_spaces=False) for g in summaries]
for abs in abstract_list:
print(abs)The output is:
torch.Size([1, 1024])
club tijuana star juan arango conjured memories luis suarez in his team 's 4-3 defeat by monterrey in the mexican league. The venezuelan icon arango sank his teeth into the shoulder of jesus zavela as his temper flared in the defeat. He was not booked by the referee but could face a heavy retrospective ban.
club tijuana star juan arango bites opponent jesus zavela in a moment of madness. arango had earlier curled in a magnificent free kick for his team to bring them level after falling 2-0 down early on in the encounter. But the 34-year-old overshadowed his goal with the bite as television cameras picked up the moment of Madness.
Juan arango bites the shoulder of jesus zavela in a moment of madness. The 34-year-old was not booked by the referee but could face a heavy retrospective ban. He had earlier curled in a magnificent free kick to bring his team level against monterrey. But the bite overshadowed his goal as television cameras picked up the moment of Madness.
juan arango bites opponent jesus zavela in a moment of madness. arango had earlier curled in a magnificent free kick for his team to bring them level after falling 2-0 down early on in the encounter. The 34-year-old overshadowed his goal with the bite as television cameras picked up the moment of Madness.
Juan arango bites the shoulder of jesus zavela in a moment of madness. The 34-year-old was not booked by the referee but could face a heavy retrospective ban. He had earlier curled in a magnificent free kick to bring his team level against monterrey.
club tijuana's Juan arango bites opponent jesus zavela. The 34-year-old was not booked but could face a heavy retrospective ban. arango had earlier curled in a magnificent free kick for his team to bring them level after falling 2-0 down early on in the encounter. But the bite overshadowed his goal as television cameras picked up the moment of madness.
juan arango bit jesus zavela as his temper flared in the defeat in mexico. arango had earlier curled in a magnificent free kick for his team to bring them level. But the 34-year-old overshadowed his goal with the bite as television cameras picked up the moment of madness.
club tijuana's Juan arango bites opponent jesus zavela. The 34-year-old was not booked but could face a heavy retrospective ban. arango had earlier curled in a magnificent free kick for his team to bring them level after falling 2-0 down early on in the encounter.
Club star juan arango bites opponent jesus zavela in mexico. arango had earlier scored a magnificent free kick to bring his team level. But the 34-year-old overshadowed his goal with the bite. He was not booked by the referee but could face a heavy retrospective ban. arangos has made 121 appearances for venezuela.
Juan arango sank his teeth into the shoulder of jesus zavela as his temper flared in the defeat. He was not booked by the referee but could face a heavy retrospective ban. The 34-year-old had earlier curled in a magnificent free kick for his team to bring them level after falling 2-0 down early on.
juan arango bites the shoulder of opponent jesus zavela as temper flares in defeat. arango had earlier curled in a magnificent free kick for his team to bring them level after falling 2-0 down early on in the encounter. The 34-year-old overshadowed his goal with the bite as television cameras picked up the moment of madness.
Club star juan arango bites opponent jesus zavela in mexico. arango had earlier scored a magnificent free kick to bring his team level. But the 34-year-old overshadowed his goal with the bite. He was not booked by the referee but could face a heavy retrospective ban.
juan arango bit an opponent in a moment of madness in mexico. arango had earlier scored a magnificent free kick for his club tijuana. But the 34-year-old overshadowed his goal with the bite. He was not booked by the referee but could face a heavy retrospective ban. arangos has made 121 appearances for venezuela.
club tijuana star juan arango bites opponent jesus zavela in a moment of madness. arango sank his teeth into the shoulder of jesUS zaVELa as his temper flared in the defeat. He was not booked by the referee but could face a heavy retrospective ban.
Club tijuana star juan arango conjured memories luis suarez in his team 's 4-3 defeat by monterrey in the mexican league. The venezuelan icon arango sank his teeth into the shoulder of jesus zavela as his temper flared in the defeat. he was not booked by the referee but could face a heavy retrospective ban.
juan arango bit an opponent in a moment of madness in mexico. arango had earlier scored a magnificent free kick for his club tijuana. But the 34-year-old overshadowed his goal with the bite. He was not booked by the referee but could face a heavy retrospective ban.The output is good. It has little difference with your result maybe because I use the uncased document as input.
I think your advice of changing transformers version is right, thanks a lot for your help! Wish you have a nice day!
Hi, yixinL7 Very beautiful work. Do you mind sharing your code about candidate generation?
I try to use the BartForConditionalGeneration model from huggingface to reproduce your results but it always generates repeat sentences in a beam.
Thanks!