search

tensorflow / tensor2tensor

Library of deep learning models and datasets designed to make deep learning more accessible and accelerate ML research.
Apache License 2.0
15.56k stars 3.51k forks source link

Question about Decoder trying to give same output as input during summarization task #959

Open prateek-kacker opened 6 years ago

prateek-kacker commented 6 years ago

I trained the algorithm with recommended parameters ( Tranformer, transformer_prepend, 200k steps). Most of the summarization task's output is either full or partial input test. What mistake am I making? What should I do to get the correct output? ... Log from output

INFO:tensorflow:Inference results INPUT: The Premier League has been the loudest opponent of the 2022 World Cup in Qatar switching to a winter tournament — staged from November 21 to December 18 — that will be rubber-stamped at the FIFA ExCo meeting in Zurich on Friday. But the richest league in world football is not expected to kick off yet again when the dates become official this week. The PL is resigned to that decision being a formality since the opinions of PL chief executive Richard Scudamore and other European leagues chiefs were virtually ignored at the task force summit in Qatar which recommended a November-December competition. ......(hidden as it was too long )

INFO:tensorflow:Inference results OUTPUT: The Premier League has been the opulenst opponent of the 2022 World Cup in Qatar switching to a winter tournament — staged from November 21 to December 18 — that will be substituted-stamped at the FIFA ExCo meeting in Wiltshire on Friday. But the richest league in world football is not expected to kick off yet again when the dates become official this week. The PL is resigned to that decision being a formality since the opinions of PL chief executive Richard WinsLibere and other European Slam chiefs were virtually ignored at the task force summit in Qatar which recommended a November-December competition. ..... (hidden as it was too long)

INFO:tensorflow:Inference results TARGET: A winter World Cup in Qatar in 2022 is expected to be confirmed on Friday. Premier League has been among loudest critics of a winter World Cup. Paul Downton and Hugh Morris to attend Global Sporting Director Summit. Jason Leonard to take up RFU president role from June .

I am using tensorflow 1.7

senarvi commented 6 years ago

I can confirm this observation. With --hparams_set=transformer_prepend the summary will be simply a copy of the input. The README says that this yields good ROUGE scores. It's true, if you truncate the input, because in terms of ROUGE scores, truncated input gives very good baseline on the CNN / Daily Mail task.

If you don't use transformer_prepend, the model will actually do something, but the results are still not good. Actually the output I see is pretty much garbage unless the inputs are truncated quite a lot.

xuekun90 commented 6 years ago

@senarvi what we want to do is summarizing the whole article. if you truncate the input, it will be no means?

senarvi commented 6 years ago

The beginning of an article is quite a good summary itself in this task. For example, just taking the three first sentences gives quite a good baseline. See for example Nallapati.

xuekun90 commented 6 years ago

@senarvi Thanks, I want to ask another question in #680, the outputs are messy codes, had you met the same issue?