search

tensorflow / nmt

TensorFlow Neural Machine Translation Tutorial
Apache License 2.0
6.37k stars 1.96k forks source link

generate BPE vocab #338

Closed yapingzhao closed 5 years ago

yapingzhao commented 6 years ago

Hi, I have a question about BPE: When I use wmt16_en_de.sh file to generate BPE vocab, why is the source language and the target language mixed?Thank you. Looking forward to your advice or answers. Best regards,

yapingzhao

kingkf commented 5 years ago

I found the same question. You get the answer? Please help me

ashim95 commented 5 years ago

@kingkf It is generally done when the source and target languages are from the same alphabet. Please refer to the original bpe paper, which has a nice discussion on joint-bpe vs independent-bpe (section 3.2)