logs_trainng_without-bpe.txt
Hi,
I have python 3.6.3 in my Debian system.
I am using the following command to train my engine:
python -m nmt.nmt \
--src=de --tgt=en \
--hparams_path=nmt/standard_hparams/wmt16_gnmt_4_layer.json \
--out_dir=/tmp/deen_gnmt \
--vocab_prefix=/tmp/wmt16/vocab.bpe.32000 \
--train_prefix=/tmp/wmt16/train.tok.clean.bpe.32000 \
--dev_prefix=/tmp/wmt16/newstest2013.tok.bpe.32000 \
--test_prefix=/tmp/wmt16/newstest2015.tok.bpe.32000
But the gnmt engine training program gets killed after the first eval step:
Here are the last few lines in the logs. Full Log File is attached for your reference.
_created eval model with fresh parameters, time 1.03s
eval dev: perplexity 76828.18, time 924s, Thu Nov 30 10:47:47 2017.
eval test: perplexity 77375.52, time 646s, Thu Nov 30 10:58:34 2017.
2017-11-30 10:58:35.226955: W tensorflow/core/kernels/lookup_util.cc:362] Table trying to initialize from file /tmp/wmt16/vocab.bpe.32000.en is already initialized.
2017-11-30 10:58:35.227568: W tensorflow/core/kernels/lookup_util.cc:362] Table trying to initialize from file /tmp/wmt16/vocab.bpe.32000.en is already initialized.
2017-11-30 10:58:35.227634: W tensorflow/core/kernels/lookup_util.cc:362] Table trying to initialize from file /tmp/wmt16/vocab.bpe.32000.de is already initialized.
created infer model with fresh parameters, time 0.91s
logs_trainng_without-bpe.txt Hi, I have python 3.6.3 in my Debian system. I am using the following command to train my engine: python -m nmt.nmt \ --src=de --tgt=en \ --hparams_path=nmt/standard_hparams/wmt16_gnmt_4_layer.json \ --out_dir=/tmp/deen_gnmt \ --vocab_prefix=/tmp/wmt16/vocab.bpe.32000 \ --train_prefix=/tmp/wmt16/train.tok.clean.bpe.32000 \ --dev_prefix=/tmp/wmt16/newstest2013.tok.bpe.32000 \ --test_prefix=/tmp/wmt16/newstest2015.tok.bpe.32000
But the gnmt engine training program gets killed after the first eval step:
Here are the last few lines in the logs. Full Log File is attached for your reference.
_created eval model with fresh parameters, time 1.03s eval dev: perplexity 76828.18, time 924s, Thu Nov 30 10:47:47 2017. eval test: perplexity 77375.52, time 646s, Thu Nov 30 10:58:34 2017. 2017-11-30 10:58:35.226955: W tensorflow/core/kernels/lookup_util.cc:362] Table trying to initialize from file /tmp/wmt16/vocab.bpe.32000.en is already initialized. 2017-11-30 10:58:35.227568: W tensorflow/core/kernels/lookup_util.cc:362] Table trying to initialize from file /tmp/wmt16/vocab.bpe.32000.en is already initialized. 2017-11-30 10:58:35.227634: W tensorflow/core/kernels/lookup_util.cc:362] Table trying to initialize from file /tmp/wmt16/vocab.bpe.32000.de is already initialized. created infer model with fresh parameters, time 0.91s
Start step 0, lr 1, Thu Nov 30 10:58:35 2017
Init train iterator, skipping 0 elements
Killed_
Any help will be very valuable to me!! Thanks