Deliberation Networks: Sequence Generation Beyond One-Pass Decoding

[kweonwooj]

Abstract

• Proposes Deliberation Network, a two-pass encoder-decoder framework with second-pass conducting deliberation
• SoTA on WMT 2014 En-Fr NMT task

Details

• Deliberation is a polishing process often seen in human cognitive behaviors during translation, writing and reading

  • Traditional Autoregressive Enc-Dec framework cannot use "future" tokens as an input.
  • Deliberation network overcomes such issue by using second-pass decoder that uses all source and one-pass decoded tokens as an input.
  • Different from Automatic Post Edit which is two-step process, whereas Deliberation Network is an one-step end-to-end framework (although there are two decoders within)

• Deliberation Network

  • 1st pass decoder generates sentence candidate
  • 2nd pass decoder uses both encoder attention, 1st pass decoder attention to generate final sentence
  • RNN (GRU, LSTM) is used as a base cell

• Training Algorithm

  • Theoretical loss function is intractable due to large space of Y
  • hence, for any src x, 1st pass decoder generates a sample y' that 2nd pass decoder learns from

• Actual Training Implementation

  • Pre-train standard encoder-decoder NMT model until convergence (took 2 weeks for EnFr)
  • for any deliberation network, the encoder is initialized by pre-trained model, both 1st & 2nd pass decoders are initialized by the decoder of pre-trained model, with the attention model for 1st pass is randomly initialized from uniform distribution of [-0.1, 0.1]
  • train deliberation network using Algorithm 1 (took 5 days for EnFr)

• Results

  • WMT En-Fr translation (small model)
  • outperforms base model, reviewer model by ~1.0 BLEU
  • Deliberation network is x2.0 slower in inference time than RNN 1-pass model
  • NIST Zh-En translation
  • always improves BLEU by +1.0 ~ 2.0
  • WMT En-Fr translation (deep model)
  • Dual Learning + Deliberation Network has +0.5 than Transformer

• Qualitative Analysis

  • awkward translation in 1st pass is fixed by 2nd pass decoder
  • egypt's middle east -> egypt says the middle east

• Future Works

  • application in improving image qualities generated by GAN
  • how to refine hidden layers of neural network in RNN, CNN
  • whether multiple passes improve sequence generation performance
  • how to speed up the inference of deliberation network

Personal Thoughts

• Great effort in implementation details : proposing new architecture, explaining the rational behind training algorithm, pre-training heuristics
• Why is this "deliberation" process needed?
  • are the original Enc-Dec framework not capable of generating perfect sentences even though source tokens hold all the context information?
  • being able to look at future tokens in decoding DOES provide additional information during decoding, but I am not fully convinced that the weakness of existing Enc-Dec framework is due to limited information in decoding step.

Link : https://papers.nips.cc/paper/6775-deliberation-networks-sequence-generation-beyond-one-pass-decoding.pdf Authors : Xia et al. 2017

[chenustc]

Do you have the code of this paper?

[kweonwooj]

@chenustc I cannot find the public code released by MS for this paper. I don't have the implementation.

[Choco31415]

@chenustc If it helps, I'm currently working on an implementation of this paper. Having trouble getting it to good results using what's specified in the paper. I'll message you somehow when things start looking good.

Update: I had stopped earlier due to not matching Microsoft's performance. I might attempt again later.

[luofuli]

@Choco31415 If you have implemented this paper, please message me too. Thank you~

[cnrpman]

@Choco31415 Please message me too, thanks!

[Huijun-Cui]

@Choco31415 Hi brother,have finished that code? I am in need of the code,would you do me a favor to send it to me? My email: cuihuijun@foxmail.com

[Hannibal046]

https://github.com/ustctf-zz/delibnet I don't know if this would help. As stated in this paper https://aclanthology.org/P19-1653.pdf, they initialize encoder and first-pass decoder with a vanilla seq2seq model, and use this model to generate samples for second-pass decoder. I'm not sure if this is the standard practice in the original Deliberation Network.