Neural Machine Translation with Gumbel-Greedy Decoding

[kweonwooj]

Abstract

• Propose Gumbel-Greedy Decoding which solves MAP problem of decoding much better
• Gumbel-Greedy Decoding makes generative network differentiable and hence trainable through SGD

Details

• Discriminator vs Generator framework

  • similar to GAN

• Gumbel-Max Trick

  • transforms sampling from a categorical distribution to an optimization problem
  • Gumble-Softmax relaxation works by relaxing discrete argmax into softmax with addtional parameter

• Result

  • marginal BLEU improvement

Personal Thoughts

• NMT like GAN
• Only marginal BLEU improvement
• No mention of inference speed

Link : https://arxiv.org/pdf/1706.07518.pdf Authors : Gu et al. 2017