Towards Decoding as Continuous Optimisation in Neural Machine Translation

[kweonwooj]

Abstract

• Continuous Optimization based Decoding

Details

• Use Exponentiated Gradient to solve re-parameterized Continuous Optimization problem

• Bidirectional Ensemble

  • use logloss of left-to-right and right-to-left inference for final inference >> not much improvement seen

• Training

  • EG with step size or momentum trains more robustly

• Result

  • Marginal BLEU improvement
  • Left-to-Right is always better

• Speed

  • Greedy : 0.02 s / sentence
  • Beam (5) : 0.07 s / sentence
  • EG (35) : 3.1 s / sentence

Personal Thoughts

• Interesting ideas all over the place, but nothing seems to show significant improvement
• Too slow in Inference time to apply

Link : https://arxiv.org/pdf/1701.02854.pdf Authors : Haffari et al. 2017