Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation (2014, cited by 8600)

[kiccho1101]

What's this paper about?

• Introduces RNN Encoder-Decoder

• One RNN encodes a sequence into a fixed length vector representation, and the other decodes the representation into another sequence.

• Model is trained to maximize p(target|source)

[kiccho1101]

Encoder

• As encoder reads each data of sequence, the hidden state changes according to the equation below.

  

• Conditional distribution of the output at each timestep t is described as below.

  

Decoder

• Hidden state of decoder changes as following equation.

  

• Therefore, conditional distribution looks like this.

  

[kiccho1101]

Training phase

• Encoder-Decoder are jointly trained to maximize the conditional log-likelihood.

[kiccho1101]

Statistical Machine Translation (SMT)

• Maximize posterior probability.

  

• f: translation

• e: source sentence

• p(e|f): translation model

• p(f): language model

[kiccho1101]

Experiment

• English/French translation task
• Compare BLEU(Bilingual Evaluation Understudy) score among Baseline, RNN, CSLM+RNN, CSLM+RNN+WP(Word Penalty)

• Best model was CSLM+RNN

[kiccho1101]

Embedding visualization