search

THUNLP-MT / THUMT

An open-source neural machine translation toolkit developed by Tsinghua Natural Language Processing Group
BSD 3-Clause "New" or "Revised" License
703 stars 197 forks source link

How about the performence? #28

Closed XiaoqingNLP closed 6 years ago

XiaoqingNLP commented 6 years ago

I reproduced this experiment with rnnsearch and seq2seq model ,I got the avarege bleu is 37.354 (rnnsearch) and 21.774(seq2seq) with zh-en corpus. I want to know how about the performence and my experiment baseline score is normal or not ? the seq2seq score is a little low .therefore ,can you given your performance ?

Playinf commented 6 years ago

The comparison should be performed on public datasets. On WMT14's En-De dataset, the BLEU score is around 14, 21, 27 for Seq2Seq, RNNsearch and Transformer respectively. I think the BLEU of your experiments is quite normal.

XiaoqingNLP commented 6 years ago

@Playinf Thanks for your answer and I have a new question about the performence. Frist question ,can you tell me your settings with implemnet model transformer? the second ,can you give me any good tips for me to Improve my performence with one or two GPU(NVIDIA TiTanX or 1080)

following performence is based wmt'17 de-en translation task

my parameters

--parameters=batch_size=3125,device_list=[1],eval_steps=5000,train_steps=100000,save_checkpoint_steps=1500,shared_embedding_and_softmax_weights=true,shared_source_target_embedding=false, update_cycle=8

this setting is follow a acl2018 paper ,but I can't get his baseline performence 34.06

my scores

I use the scripts multi-bleu-detoke.perl with case-insensitive and got the bleu 31.79 .

XiaoqingNLP commented 6 years ago

new question about the performence

I have check the thumt code of multi_gpu part ,I find it is similar to tensor2tensor ,and I got the score is different with single GPU and Multi-GPU performence with different code. thumt single GPU differ with multiGPU about :0.5 tensor2tensor single GPU differ with multiGPU about:2 can you explain with me ?please ,thank you for your kindly answers and work.

thumt baseline score

params/bpe nist02 nist03 nist04 nist05 nist08 nist-avg
141-transformer-1gpu nobpe 42.87 41.24 43.81 41.53 32.03 40.296
141-transformer/2-gpu/ model 43.33 41.35 44.04 42.26 32.88 40.772

tensor2tensor baseline score

transformer/muti-GPU/params nist02 nist03 nist04 nist05 nist08 nist-avg
1-gpu/baseline-141 42.02 41.15 43.93 41.57 31.99 40.132
transformer/2-gpu/baseline-141 44.07 43.34 45.38 43.44 34.06 42.058

PS:141 means GPU Nvidia Titan X