tensorflow / nmt

TensorFlow Neural Machine Translation Tutorial
Apache License 2.0
6.39k stars 1.96k forks source link

Tensorflow gnmt engine training gets killed after the first eval step #197

Open Sabyasachi18 opened 6 years ago

Sabyasachi18 commented 6 years ago

logs_trainng_without-bpe.txt Hi, I have python 3.6.3 in my Debian system. I am using the following command to train my engine: python -m nmt.nmt \ --src=de --tgt=en \ --hparams_path=nmt/standard_hparams/wmt16_gnmt_4_layer.json \ --out_dir=/tmp/deen_gnmt \ --vocab_prefix=/tmp/wmt16/vocab.bpe.32000 \ --train_prefix=/tmp/wmt16/train.tok.clean.bpe.32000 \ --dev_prefix=/tmp/wmt16/newstest2013.tok.bpe.32000 \ --test_prefix=/tmp/wmt16/newstest2015.tok.bpe.32000

But the gnmt engine training program gets killed after the first eval step:

Here are the last few lines in the logs. Full Log File is attached for your reference.

_created eval model with fresh parameters, time 1.03s eval dev: perplexity 76828.18, time 924s, Thu Nov 30 10:47:47 2017. eval test: perplexity 77375.52, time 646s, Thu Nov 30 10:58:34 2017. 2017-11-30 10:58:35.226955: W tensorflow/core/kernels/lookup_util.cc:362] Table trying to initialize from file /tmp/wmt16/vocab.bpe.32000.en is already initialized. 2017-11-30 10:58:35.227568: W tensorflow/core/kernels/lookup_util.cc:362] Table trying to initialize from file /tmp/wmt16/vocab.bpe.32000.en is already initialized. 2017-11-30 10:58:35.227634: W tensorflow/core/kernels/lookup_util.cc:362] Table trying to initialize from file /tmp/wmt16/vocab.bpe.32000.de is already initialized. created infer model with fresh parameters, time 0.91s

Start step 0, lr 1, Thu Nov 30 10:58:35 2017

Init train iterator, skipping 0 elements

Killed_

Any help will be very valuable to me!! Thanks

Sabyasachi18 commented 6 years ago

Hi, This issue was solved after I downgraded my python version from 3.6.3 to 3.5.3. Tensforflow version 1.5.0-dev20171115 THank You