Towards Neural Phrase-based Machine Translation - Githubissues

kweonwooj / papers

summary of ML papers I've read

319 stars 34 forks source link

Towards Neural Phrase-based Machine Translation #85

Open kweonwooj opened 6 years ago

kweonwooj commented 6 years ago

Abstract

propose a phrase-based NMT architecture using SWAN and soft local reordering layer
SoTA on small NMT dataset
- IWSLT14 EnDe/DeEn, IWSLT15 EnVi

Details

Introduction

NPMT combines advantages of SMT and NMT
- SMT : phrase-based SMT approach consistently outperformed word-based
- NMT : work surprisingly well without explicit phrase treatment
SWAN requires monotonic alignment between inputs and outputs -> introduce soft local reordering layer

Neural Phrase-based Machine Translation

screen shot 2018-02-06 at 4 25 36 pm

Architecture
- word embedding lookup table
- soft reordering layer by gating mechanism
- each gating value is obtained by looking at neighbor tokens within a window size (7 is hyper-parameter)
- Bi-directional RNN layer (can be GRU, LSTM, Convolutional Block etc)
- SWAN layer : Sleep and WAke layer where each phrase is decoded separately. details in here

Results

IWSLT14 German-English
- SoTA compared to attention-based models and Actor-Critic model
- score improves with LM objective
IWSLT14 English-German
IWSLT15 English-Vietnamese

Sample

German -> English translation sample
Visualization of Reordering Layer
- window size 7
- English -> German
- ? has gate activation in übersetzen ?

Personal Thoughts

Soft Reordering layer is a creative layer! 👍
- it's surprising that source embeddings can learn the word order that is inherent in target language without information flowing from target labels
wish to see Speed, Memory, Performance on WMT14 datasets

Link : https://arxiv.org/pdf/1706.05565.pdf Authors : Huang et al. 2018