search

neccam / nslt

Neural Sign Language Translation (CVPR'18)
Apache License 2.0
149 stars 41 forks source link

Another error during training (python 2.7) #13

Open vbelissen opened 5 years ago

vbelissen commented 5 years ago

Here is the command I run: python -m nmt --src=sign --tgt=de --train_prefix=../Data/phoenix2014T.train --dev_prefix=../Data/phoenix2014T.dev --test_prefix=../Data/phoenix2014T.test --out_dir=../test_out/ --vocab_prefix=../Data/phoenix2014T.vocab --source_reverse=True --num_units=1000 --num_layers=4 --num_train_steps=150000 --residual=True --attention=luong --base_gpu=0 --unit_type=gru

Here is the error I get, with python 2.7 and TF 1.3 (or 1.4.1):

# log_file=../test_out/log_1552151343
  created train model with fresh parameters, time 0.98s
  created infer model with fresh parameters, time 0.23s
  # 301
utils/nmt_utils.py:92: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
  if tgt_eos and tgt_eos in output:
    src: /localHD/phoenix/PHOENIX-2014-T-release-v3/PHOENIX-2014-T/features/fullFrame-227x227px/dev/20June_2011_Monday_heute-6514/
    ref: und eher wechselhaft geht es mit unserem wetter auch weiter .
    nmt: südwind südwind nämlich nämlich nämlich nämlich wochenende wochenende wochenende wochenende wochenende wochenende wochenende wochenende wochenende wochenende wochenende düsseldorf düsseldorf düsseldorf nieselregen nieselregen nieselregen tauwetters tauwetters tauwetters tauwetters tauwetters tauwetters tauwetters tauwetters tauwetters tauwetters tauwetters tauwetters tauwetters tauwetters tauwetters tauwetters tauwetters
  created eval model with fresh parameters, time 0.17s
Traceback (most recent call last):
  File "/people/belissen/anaconda3/envs/py27_nslt/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/people/belissen/anaconda3/envs/py27_nslt/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/people/belissen/Python/nslt-master/nslt/nmt.py", line 378, in <module>
    tf.app.run(main=main, argv=[sys.argv[0]] + unparsed)
  File "/people/belissen/anaconda3/envs/py27_nslt/lib/python2.7/site-packages/tensorflow/python/platform/app.py", line 48, in run
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "/people/belissen/Python/nslt-master/nslt/nmt.py", line 368, in main
    run_main(FLAGS, default_hparams, train_fn, inference_fn)
  File "/people/belissen/Python/nslt-master/nslt/nmt.py", line 361, in run_main
    train_fn(hparams, target_session=target_session)
  File "train.py", line 265, in train
    run_full_eval(model_dir, infer_model, infer_sess, eval_model, eval_sess, hparams, summary_writer, sample_src_data, sample_tgt_data)
  File "train.py", line 189, in run_full_eval
    dev_ppl, test_ppl = run_internal_eval(eval_model, eval_sess, model_dir, hparams, summary_writer)
  File "train.py", line 148, in run_internal_eval
    dev_ppl = _internal_eval(loaded_eval_model, global_step, eval_sess, eval_model.iterator, dev_eval_iterator_feed_dict, summary_writer, "dev")
  File "train.py", line 405, in _internal_eval
    ppl = model_helper.compute_perplexity(model, sess, label)
  File "model_helper.py", line 215, in compute_perplexity
    loss, predict_count, batch_size = model.eval(sess)
  File "model.py", line 181, in eval
    self.batch_size])
  File "/people/belissen/anaconda3/envs/py27_nslt/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 889, in run
    run_metadata_ptr)
  File "/people/belissen/anaconda3/envs/py27_nslt/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1120, in _run
    feed_dict_tensor, options, run_metadata)
  File "/people/belissen/anaconda3/envs/py27_nslt/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1317, in _do_run
    options, run_metadata)
  File "/people/belissen/anaconda3/envs/py27_nslt/lib/python2.7/site-packages/tensorflow/python/client/session.py", line 1336, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.UnknownError: exceptions.AttributeError: 'NoneType' object has no attribute 'astype'
         [[Node: PyFunc = PyFunc[Tin=[DT_STRING, DT_BOOL], Tout=[DT_FLOAT], token="pyfunc_3"](arg0, PyFunc/input_1)]]
         [[Node: IteratorGetNext = IteratorGetNext[output_shapes=[[300,227,227,3], [1,?], [1,?], [1], [1]], output_types=[DT_FLOAT, DT_INT32, DT_INT32, DT_INT32, DT_INT32], _device="/job:localhost/replica:0/task:0/device:CPU:0"](Iterator)]]
neccam commented 5 years ago

I have just tested the code again and it runs without any problem for me when I ran the script you have provided. 

python -m nmt --src=sign --tgt=de --train_prefix=../Data/phoenix2014T.train --dev_prefix=../Data/phoenix2014T.dev --test_prefix=../Data/phoenix2014T.test --out_dir=../test_out/ --vocab_prefix=../Data/phoenix2014T.vocab --source_reverse=True --num_units=1000 --num_layers=4 --num_train_steps=150000 --residual=True --attention=luong --base_gpu=0 --unit_type=gru

I used a virtual environment with python 2.7.14 and tensorflow 1.4.0-rc1. As cuda I have 8.0 and cudnn 6.0. See the list of installed packages in my environment below:

backports.weakref         1.0.post1                 <pip>
bleach                    1.5.0                     <pip>
ca-certificates           2017.08.26           h1d4fec5_0  
certifi                   2018.1.18                py27_0  
enum34                    1.1.6                     <pip>
funcsigs                  1.0.2                     <pip>
futures                   3.2.0                     <pip>
html5lib                  0.9999999                 <pip>
intel-openmp              2018.0.0                      8  
libedit                   3.1                  heed3624_0  
libffi                    3.2.1                hd88cf55_4  
libgcc-ng                 7.2.0                h7cc24e2_2  
libgfortran-ng            7.2.0                hdf63c60_3  
libstdcxx-ng              7.2.0                h7a57d05_2  
Markdown                  2.6.11                    <pip>
mkl                       2018.0.2                      1  
mkl_fft                   1.0.1            py27h3010b51_0  
mkl_random                1.0.1            py27h629b387_0  
mock                      2.0.0                     <pip>
ncurses                   6.0                  h9df7e31_2  
numpy                     1.14.2           py27hdbf6ddf_1  
numpy                     1.14.0                    <pip>
opencv                    2.4.11                 nppy27_0    menpo
openssl                   1.0.2n               hb7f436b_0  
pbr                       3.1.1                     <pip>
pip                       9.0.1            py27ha730c48_4  
protobuf                  3.5.1                     <pip>
python                    2.7.14              h1571d57_29  
readline                  7.0                  ha6073c6_4  
scikit-learn              0.19.1                    <pip>
scipy                     1.0.0                     <pip>
setuptools                38.5.0                    <pip>
setuptools                38.4.0                   py27_0  
six                       1.11.0                    <pip>
sklearn                   0.0                       <pip>
sqlite                    3.22.0               h1bed415_0  
tensorflow-gpu            1.4.0rc1                  <pip>
tensorflow-tensorboard    0.4.0                     <pip>
tk                        8.6.7                hc745277_3  
Werkzeug                  0.14.1                    <pip>
wheel                     0.30.0           py27h2bc6bb2_1  
wheel                     0.30.0                    <pip>
zlib                      1.2.11               ha838bed_2

I see from your log that you have created a virtual environment with python 2.7 using Anaconda 3. My suggestion would be to install Anaconda 2 and create your virtual environment using it instead.

And here is the log:

# Job id 0
# Set random seed to 285
# hparams:
  src=sign
  tgt=de
  train_prefix=../Data/phoenix2014T.train
  dev_prefix=../Data/phoenix2014T.dev
  test_prefix=../Data/phoenix2014T.test
  out_dir=../test_out/
# Vocab file ../Data/phoenix2014T.vocab.de exists
  saving hparams to ../test_out/hparams
  saving hparams to ../test_out/best_bleu/hparams
  attention=luong
  attention_architecture=standard
  base_gpu=0
  batch_size=1
  beam_width=3
  best_bleu=0
  best_bleu_dir=../test_out/best_bleu
  bpe_delimiter=None
  colocate_gradients_with_ops=True
  decay_factor=0.98
  decay_steps=10000
  dev_prefix=../Data/phoenix2014T.dev
  dropout=0.2
  encoder_type=uni
  eos=</s>
  epoch_step=0
  eval_on_fly=True
  forget_bias=1.0
  infer_batch_size=32
  init_op=glorot_normal
  init_weight=0.1
  learning_rate=1e-05
  length_penalty_weight=0.0
  log_device_placement=False
  max_gradient_norm=5.0
  max_train=0
  metrics=['bleu']
  num_buckets=0
  num_embeddings_partitions=0
  num_gpus=1
  num_layers=4
  num_residual_layers=3
  num_train_steps=150000
  num_units=1000
  optimizer=adam
  out_dir=../test_out/
  pass_hidden_state=True
  random_seed=285
  residual=True
  snapshot_interval=1000
  sos=<s>
  source_reverse=True
  src=sign
  src_max_len=300
  src_max_len_infer=300
  start_decay_step=0
  steps_per_external_eval=None
  steps_per_stats=100
  test_prefix=../Data/phoenix2014T.test
  tgt=de
  tgt_max_len=50
  tgt_max_len_infer=None
  tgt_vocab_file=../Data/phoenix2014T.vocab.de
  tgt_vocab_size=2891
  time_major=True
  train_prefix=../Data/phoenix2014T.train
  unit_type=gru
  vocab_prefix=../Data/phoenix2014T.vocab
WARNING:tensorflow:From train.py:59: __init__ (from tensorflow.contrib.data.python.ops.readers) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.TextLineDataset`.
WARNING:tensorflow:From utils/iterator_utils.py:111: zip (from tensorflow.contrib.data.python.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.zip()`.
WARNING:tensorflow:From utils/iterator_utils.py:123: calling map (from tensorflow.contrib.data.python.ops.dataset_ops) with num_threads is deprecated and will be removed in a future version.
Instructions for updating:
Replace `num_threads=T` with `num_parallel_calls=T`. Replace `output_buffer_size=N` with `ds.prefetch(N)` on the returned dataset.
WARNING:tensorflow:From utils/iterator_utils.py:123: calling map (from tensorflow.contrib.data.python.ops.dataset_ops) with output_buffer_size is deprecated and will be removed in a future version.
Instructions for updating:
Replace `num_threads=T` with `num_parallel_calls=T`. Replace `output_buffer_size=N` with `ds.prefetch(N)` on the returned dataset.
# creating train graph ...
  num_layers = 4, num_residual_layers=3
  cell 0  GRU  DropoutWrapper, dropout=0.2   DeviceWrapper, device=/gpu:0
  cell 1  GRU  DropoutWrapper, dropout=0.2   ResidualWrapper  DeviceWrapper, device=/gpu:0
  cell 2  GRU  DropoutWrapper, dropout=0.2   ResidualWrapper  DeviceWrapper, device=/gpu:0
  cell 3  GRU  DropoutWrapper, dropout=0.2   ResidualWrapper  DeviceWrapper, device=/gpu:0
  cell 0  GRU  DropoutWrapper, dropout=0.2   DeviceWrapper, device=/gpu:0
  cell 1  GRU  DropoutWrapper, dropout=0.2   ResidualWrapper  DeviceWrapper, device=/gpu:0
  cell 2  GRU  DropoutWrapper, dropout=0.2   ResidualWrapper  DeviceWrapper, device=/gpu:0
  cell 3  GRU  DropoutWrapper, dropout=0.2   ResidualWrapper  DeviceWrapper, device=/gpu:0
  start_decay_step=0, learning_rate=1e-05, decay_steps 10000, decay_factor 0.98
# Trainable variables
  conv1/weights:0, (11, 11, 3, 96), /device:GPU:0
  conv1/biases:0, (96,), /device:GPU:0
  conv2/weights:0, (5, 5, 48, 256), /device:GPU:0
  conv2/biases:0, (256,), /device:GPU:0
  conv3/weights:0, (3, 3, 256, 384), /device:GPU:0
  conv3/biases:0, (384,), /device:GPU:0
  conv4/weights:0, (3, 3, 192, 384), /device:GPU:0
  conv4/biases:0, (384,), /device:GPU:0
  conv5/weights:0, (3, 3, 192, 256), /device:GPU:0
  conv5/biases:0, (256,), /device:GPU:0
  fc6/weights:0, (9216, 4096), /device:GPU:0
  fc6/biases:0, (4096,), /device:GPU:0
  fc7/weights:0, (4096, 4096), /device:GPU:0
  fc7/biases:0, (4096,), /device:GPU:0
  embeddings/decoder/embedding_decoder:0, (2891, 1000), /device:GPU:0
  dynamic_seq2seq/encoder/rnn/multi_rnn_cell/cell_0/gru_cell/gates/kernel:0, (5096, 2000), /device:GPU:0
  dynamic_seq2seq/encoder/rnn/multi_rnn_cell/cell_0/gru_cell/gates/bias:0, (2000,), /device:GPU:0
  dynamic_seq2seq/encoder/rnn/multi_rnn_cell/cell_0/gru_cell/candidate/kernel:0, (5096, 1000), /device:GPU:0
  dynamic_seq2seq/encoder/rnn/multi_rnn_cell/cell_0/gru_cell/candidate/bias:0, (1000,), /device:GPU:0
  dynamic_seq2seq/encoder/rnn/multi_rnn_cell/cell_1/gru_cell/gates/kernel:0, (2000, 2000), /device:GPU:0
  dynamic_seq2seq/encoder/rnn/multi_rnn_cell/cell_1/gru_cell/gates/bias:0, (2000,), /device:GPU:0
  dynamic_seq2seq/encoder/rnn/multi_rnn_cell/cell_1/gru_cell/candidate/kernel:0, (2000, 1000), /device:GPU:0
  dynamic_seq2seq/encoder/rnn/multi_rnn_cell/cell_1/gru_cell/candidate/bias:0, (1000,), /device:GPU:0
  dynamic_seq2seq/encoder/rnn/multi_rnn_cell/cell_2/gru_cell/gates/kernel:0, (2000, 2000), /device:GPU:0
  dynamic_seq2seq/encoder/rnn/multi_rnn_cell/cell_2/gru_cell/gates/bias:0, (2000,), /device:GPU:0
  dynamic_seq2seq/encoder/rnn/multi_rnn_cell/cell_2/gru_cell/candidate/kernel:0, (2000, 1000), /device:GPU:0
  dynamic_seq2seq/encoder/rnn/multi_rnn_cell/cell_2/gru_cell/candidate/bias:0, (1000,), /device:GPU:0
  dynamic_seq2seq/encoder/rnn/multi_rnn_cell/cell_3/gru_cell/gates/kernel:0, (2000, 2000), /device:GPU:0
  dynamic_seq2seq/encoder/rnn/multi_rnn_cell/cell_3/gru_cell/gates/bias:0, (2000,), /device:GPU:0
  dynamic_seq2seq/encoder/rnn/multi_rnn_cell/cell_3/gru_cell/candidate/kernel:0, (2000, 1000), /device:GPU:0
  dynamic_seq2seq/encoder/rnn/multi_rnn_cell/cell_3/gru_cell/candidate/bias:0, (1000,), /device:GPU:0
  dynamic_seq2seq/decoder/memory_layer/kernel:0, (1000, 1000), /device:GPU:0
  dynamic_seq2seq/decoder/attention/multi_rnn_cell/cell_0/gru_cell/gates/kernel:0, (3000, 2000), /device:GPU:0
  dynamic_seq2seq/decoder/attention/multi_rnn_cell/cell_0/gru_cell/gates/bias:0, (2000,), /device:GPU:0
  dynamic_seq2seq/decoder/attention/multi_rnn_cell/cell_0/gru_cell/candidate/kernel:0, (3000, 1000), /device:GPU:0
  dynamic_seq2seq/decoder/attention/multi_rnn_cell/cell_0/gru_cell/candidate/bias:0, (1000,), /device:GPU:0
  dynamic_seq2seq/decoder/attention/multi_rnn_cell/cell_1/gru_cell/gates/kernel:0, (2000, 2000), /device:GPU:0
  dynamic_seq2seq/decoder/attention/multi_rnn_cell/cell_1/gru_cell/gates/bias:0, (2000,), /device:GPU:0
  dynamic_seq2seq/decoder/attention/multi_rnn_cell/cell_1/gru_cell/candidate/kernel:0, (2000, 1000), /device:GPU:0
  dynamic_seq2seq/decoder/attention/multi_rnn_cell/cell_1/gru_cell/candidate/bias:0, (1000,), /device:GPU:0
  dynamic_seq2seq/decoder/attention/multi_rnn_cell/cell_2/gru_cell/gates/kernel:0, (2000, 2000), /device:GPU:0
  dynamic_seq2seq/decoder/attention/multi_rnn_cell/cell_2/gru_cell/gates/bias:0, (2000,), /device:GPU:0
  dynamic_seq2seq/decoder/attention/multi_rnn_cell/cell_2/gru_cell/candidate/kernel:0, (2000, 1000), /device:GPU:0
  dynamic_seq2seq/decoder/attention/multi_rnn_cell/cell_2/gru_cell/candidate/bias:0, (1000,), /device:GPU:0
  dynamic_seq2seq/decoder/attention/multi_rnn_cell/cell_3/gru_cell/gates/kernel:0, (2000, 2000), /device:GPU:0
  dynamic_seq2seq/decoder/attention/multi_rnn_cell/cell_3/gru_cell/gates/bias:0, (2000,), /device:GPU:0
  dynamic_seq2seq/decoder/attention/multi_rnn_cell/cell_3/gru_cell/candidate/kernel:0, (2000, 1000), /device:GPU:0
  dynamic_seq2seq/decoder/attention/multi_rnn_cell/cell_3/gru_cell/candidate/bias:0, (1000,), /device:GPU:0
  dynamic_seq2seq/decoder/attention/attention_layer/kernel:0, (2000, 1000), /device:GPU:0
  dynamic_seq2seq/decoder/output_projection/kernel:0, (1000, 2891), /device:GPU:0
# creating eval graph ...
  num_layers = 4, num_residual_layers=3
  cell 0  GRU  DeviceWrapper, device=/gpu:0
  cell 1  GRU  ResidualWrapper  DeviceWrapper, device=/gpu:0
  cell 2  GRU  ResidualWrapper  DeviceWrapper, device=/gpu:0
  cell 3  GRU  ResidualWrapper  DeviceWrapper, device=/gpu:0
  cell 0  GRU  DeviceWrapper, device=/gpu:0
  cell 1  GRU  ResidualWrapper  DeviceWrapper, device=/gpu:0
  cell 2  GRU  ResidualWrapper  DeviceWrapper, device=/gpu:0
  cell 3  GRU  ResidualWrapper  DeviceWrapper, device=/gpu:0
  start_decay_step=0, learning_rate=1e-05, decay_steps 10000, decay_factor 0.98
# Trainable variables
  conv1/weights:0, (11, 11, 3, 96), /device:GPU:0
  conv1/biases:0, (96,), /device:GPU:0
  conv2/weights:0, (5, 5, 48, 256), /device:GPU:0
  conv2/biases:0, (256,), /device:GPU:0
  conv3/weights:0, (3, 3, 256, 384), /device:GPU:0
  conv3/biases:0, (384,), /device:GPU:0
  conv4/weights:0, (3, 3, 192, 384), /device:GPU:0
  conv4/biases:0, (384,), /device:GPU:0
  conv5/weights:0, (3, 3, 192, 256), /device:GPU:0
  conv5/biases:0, (256,), /device:GPU:0
  fc6/weights:0, (9216, 4096), /device:GPU:0
  fc6/biases:0, (4096,), /device:GPU:0
  fc7/weights:0, (4096, 4096), /device:GPU:0
  fc7/biases:0, (4096,), /device:GPU:0
  embeddings/decoder/embedding_decoder:0, (2891, 1000), 
  dynamic_seq2seq/encoder/rnn/multi_rnn_cell/cell_0/gru_cell/gates/kernel:0, (5096, 2000), /device:GPU:0
  dynamic_seq2seq/encoder/rnn/multi_rnn_cell/cell_0/gru_cell/gates/bias:0, (2000,), /device:GPU:0
  dynamic_seq2seq/encoder/rnn/multi_rnn_cell/cell_0/gru_cell/candidate/kernel:0, (5096, 1000), /device:GPU:0
  dynamic_seq2seq/encoder/rnn/multi_rnn_cell/cell_0/gru_cell/candidate/bias:0, (1000,), /device:GPU:0
  dynamic_seq2seq/encoder/rnn/multi_rnn_cell/cell_1/gru_cell/gates/kernel:0, (2000, 2000), /device:GPU:0
  dynamic_seq2seq/encoder/rnn/multi_rnn_cell/cell_1/gru_cell/gates/bias:0, (2000,), /device:GPU:0
  dynamic_seq2seq/encoder/rnn/multi_rnn_cell/cell_1/gru_cell/candidate/kernel:0, (2000, 1000), /device:GPU:0
  dynamic_seq2seq/encoder/rnn/multi_rnn_cell/cell_1/gru_cell/candidate/bias:0, (1000,), /device:GPU:0
  dynamic_seq2seq/encoder/rnn/multi_rnn_cell/cell_2/gru_cell/gates/kernel:0, (2000, 2000), /device:GPU:0
  dynamic_seq2seq/encoder/rnn/multi_rnn_cell/cell_2/gru_cell/gates/bias:0, (2000,), /device:GPU:0
  dynamic_seq2seq/encoder/rnn/multi_rnn_cell/cell_2/gru_cell/candidate/kernel:0, (2000, 1000), /device:GPU:0
  dynamic_seq2seq/encoder/rnn/multi_rnn_cell/cell_2/gru_cell/candidate/bias:0, (1000,), /device:GPU:0
  dynamic_seq2seq/encoder/rnn/multi_rnn_cell/cell_3/gru_cell/gates/kernel:0, (2000, 2000), /device:GPU:0
  dynamic_seq2seq/encoder/rnn/multi_rnn_cell/cell_3/gru_cell/gates/bias:0, (2000,), /device:GPU:0
  dynamic_seq2seq/encoder/rnn/multi_rnn_cell/cell_3/gru_cell/candidate/kernel:0, (2000, 1000), /device:GPU:0
  dynamic_seq2seq/encoder/rnn/multi_rnn_cell/cell_3/gru_cell/candidate/bias:0, (1000,), /device:GPU:0
  dynamic_seq2seq/decoder/memory_layer/kernel:0, (1000, 1000), 
  dynamic_seq2seq/decoder/attention/multi_rnn_cell/cell_0/gru_cell/gates/kernel:0, (3000, 2000), /device:GPU:0
  dynamic_seq2seq/decoder/attention/multi_rnn_cell/cell_0/gru_cell/gates/bias:0, (2000,), /device:GPU:0
  dynamic_seq2seq/decoder/attention/multi_rnn_cell/cell_0/gru_cell/candidate/kernel:0, (3000, 1000), /device:GPU:0
  dynamic_seq2seq/decoder/attention/multi_rnn_cell/cell_0/gru_cell/candidate/bias:0, (1000,), /device:GPU:0
  dynamic_seq2seq/decoder/attention/multi_rnn_cell/cell_1/gru_cell/gates/kernel:0, (2000, 2000), /device:GPU:0
  dynamic_seq2seq/decoder/attention/multi_rnn_cell/cell_1/gru_cell/gates/bias:0, (2000,), /device:GPU:0
  dynamic_seq2seq/decoder/attention/multi_rnn_cell/cell_1/gru_cell/candidate/kernel:0, (2000, 1000), /device:GPU:0
  dynamic_seq2seq/decoder/attention/multi_rnn_cell/cell_1/gru_cell/candidate/bias:0, (1000,), /device:GPU:0
  dynamic_seq2seq/decoder/attention/multi_rnn_cell/cell_2/gru_cell/gates/kernel:0, (2000, 2000), /device:GPU:0
  dynamic_seq2seq/decoder/attention/multi_rnn_cell/cell_2/gru_cell/gates/bias:0, (2000,), /device:GPU:0
  dynamic_seq2seq/decoder/attention/multi_rnn_cell/cell_2/gru_cell/candidate/kernel:0, (2000, 1000), /device:GPU:0
  dynamic_seq2seq/decoder/attention/multi_rnn_cell/cell_2/gru_cell/candidate/bias:0, (1000,), /device:GPU:0
  dynamic_seq2seq/decoder/attention/multi_rnn_cell/cell_3/gru_cell/gates/kernel:0, (2000, 2000), /device:GPU:0
  dynamic_seq2seq/decoder/attention/multi_rnn_cell/cell_3/gru_cell/gates/bias:0, (2000,), /device:GPU:0
  dynamic_seq2seq/decoder/attention/multi_rnn_cell/cell_3/gru_cell/candidate/kernel:0, (2000, 1000), /device:GPU:0
  dynamic_seq2seq/decoder/attention/multi_rnn_cell/cell_3/gru_cell/candidate/bias:0, (1000,), /device:GPU:0
  dynamic_seq2seq/decoder/attention/attention_layer/kernel:0, (2000, 1000), /device:GPU:0
  dynamic_seq2seq/decoder/output_projection/kernel:0, (1000, 2891), /device:GPU:0
WARNING:tensorflow:From inference.py:57: from_tensor_slices (from tensorflow.contrib.data.python.ops.dataset_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.data.Dataset.from_tensor_slices()`.
# creating infer graph ...
  num_layers = 4, num_residual_layers=3
  cell 0  GRU  DeviceWrapper, device=/gpu:0
  cell 1  GRU  ResidualWrapper  DeviceWrapper, device=/gpu:0
  cell 2  GRU  ResidualWrapper  DeviceWrapper, device=/gpu:0
  cell 3  GRU  ResidualWrapper  DeviceWrapper, device=/gpu:0
  cell 0  GRU  DeviceWrapper, device=/gpu:0
  cell 1  GRU  ResidualWrapper  DeviceWrapper, device=/gpu:0
  cell 2  GRU  ResidualWrapper  DeviceWrapper, device=/gpu:0
  cell 3  GRU  ResidualWrapper  DeviceWrapper, device=/gpu:0
  start_decay_step=0, learning_rate=1e-05, decay_steps 10000, decay_factor 0.98
# Trainable variables
  conv1/weights:0, (11, 11, 3, 96), /device:GPU:0
  conv1/biases:0, (96,), /device:GPU:0
  conv2/weights:0, (5, 5, 48, 256), /device:GPU:0
  conv2/biases:0, (256,), /device:GPU:0
  conv3/weights:0, (3, 3, 256, 384), /device:GPU:0
  conv3/biases:0, (384,), /device:GPU:0
  conv4/weights:0, (3, 3, 192, 384), /device:GPU:0
  conv4/biases:0, (384,), /device:GPU:0
  conv5/weights:0, (3, 3, 192, 256), /device:GPU:0
  conv5/biases:0, (256,), /device:GPU:0
  fc6/weights:0, (9216, 4096), /device:GPU:0
  fc6/biases:0, (4096,), /device:GPU:0
  fc7/weights:0, (4096, 4096), /device:GPU:0
  fc7/biases:0, (4096,), /device:GPU:0
  embeddings/decoder/embedding_decoder:0, (2891, 1000), 
  dynamic_seq2seq/encoder/rnn/multi_rnn_cell/cell_0/gru_cell/gates/kernel:0, (5096, 2000), /device:GPU:0
  dynamic_seq2seq/encoder/rnn/multi_rnn_cell/cell_0/gru_cell/gates/bias:0, (2000,), /device:GPU:0
  dynamic_seq2seq/encoder/rnn/multi_rnn_cell/cell_0/gru_cell/candidate/kernel:0, (5096, 1000), /device:GPU:0
  dynamic_seq2seq/encoder/rnn/multi_rnn_cell/cell_0/gru_cell/candidate/bias:0, (1000,), /device:GPU:0
  dynamic_seq2seq/encoder/rnn/multi_rnn_cell/cell_1/gru_cell/gates/kernel:0, (2000, 2000), /device:GPU:0
  dynamic_seq2seq/encoder/rnn/multi_rnn_cell/cell_1/gru_cell/gates/bias:0, (2000,), /device:GPU:0
  dynamic_seq2seq/encoder/rnn/multi_rnn_cell/cell_1/gru_cell/candidate/kernel:0, (2000, 1000), /device:GPU:0
  dynamic_seq2seq/encoder/rnn/multi_rnn_cell/cell_1/gru_cell/candidate/bias:0, (1000,), /device:GPU:0
  dynamic_seq2seq/encoder/rnn/multi_rnn_cell/cell_2/gru_cell/gates/kernel:0, (2000, 2000), /device:GPU:0
  dynamic_seq2seq/encoder/rnn/multi_rnn_cell/cell_2/gru_cell/gates/bias:0, (2000,), /device:GPU:0
  dynamic_seq2seq/encoder/rnn/multi_rnn_cell/cell_2/gru_cell/candidate/kernel:0, (2000, 1000), /device:GPU:0
  dynamic_seq2seq/encoder/rnn/multi_rnn_cell/cell_2/gru_cell/candidate/bias:0, (1000,), /device:GPU:0
  dynamic_seq2seq/encoder/rnn/multi_rnn_cell/cell_3/gru_cell/gates/kernel:0, (2000, 2000), /device:GPU:0
  dynamic_seq2seq/encoder/rnn/multi_rnn_cell/cell_3/gru_cell/gates/bias:0, (2000,), /device:GPU:0
  dynamic_seq2seq/encoder/rnn/multi_rnn_cell/cell_3/gru_cell/candidate/kernel:0, (2000, 1000), /device:GPU:0
  dynamic_seq2seq/encoder/rnn/multi_rnn_cell/cell_3/gru_cell/candidate/bias:0, (1000,), /device:GPU:0
  dynamic_seq2seq/decoder/memory_layer/kernel:0, (1000, 1000), 
  dynamic_seq2seq/decoder/attention/multi_rnn_cell/cell_0/gru_cell/gates/kernel:0, (3000, 2000), /device:GPU:0
  dynamic_seq2seq/decoder/attention/multi_rnn_cell/cell_0/gru_cell/gates/bias:0, (2000,), /device:GPU:0
  dynamic_seq2seq/decoder/attention/multi_rnn_cell/cell_0/gru_cell/candidate/kernel:0, (3000, 1000), /device:GPU:0
  dynamic_seq2seq/decoder/attention/multi_rnn_cell/cell_0/gru_cell/candidate/bias:0, (1000,), /device:GPU:0
  dynamic_seq2seq/decoder/attention/multi_rnn_cell/cell_1/gru_cell/gates/kernel:0, (2000, 2000), /device:GPU:0
  dynamic_seq2seq/decoder/attention/multi_rnn_cell/cell_1/gru_cell/gates/bias:0, (2000,), /device:GPU:0
  dynamic_seq2seq/decoder/attention/multi_rnn_cell/cell_1/gru_cell/candidate/kernel:0, (2000, 1000), /device:GPU:0
  dynamic_seq2seq/decoder/attention/multi_rnn_cell/cell_1/gru_cell/candidate/bias:0, (1000,), /device:GPU:0
  dynamic_seq2seq/decoder/attention/multi_rnn_cell/cell_2/gru_cell/gates/kernel:0, (2000, 2000), /device:GPU:0
  dynamic_seq2seq/decoder/attention/multi_rnn_cell/cell_2/gru_cell/gates/bias:0, (2000,), /device:GPU:0
  dynamic_seq2seq/decoder/attention/multi_rnn_cell/cell_2/gru_cell/candidate/kernel:0, (2000, 1000), /device:GPU:0
  dynamic_seq2seq/decoder/attention/multi_rnn_cell/cell_2/gru_cell/candidate/bias:0, (1000,), /device:GPU:0
  dynamic_seq2seq/decoder/attention/multi_rnn_cell/cell_3/gru_cell/gates/kernel:0, (2000, 2000), /device:GPU:0
  dynamic_seq2seq/decoder/attention/multi_rnn_cell/cell_3/gru_cell/gates/bias:0, (2000,), /device:GPU:0
  dynamic_seq2seq/decoder/attention/multi_rnn_cell/cell_3/gru_cell/candidate/kernel:0, (2000, 1000), /device:GPU:0
  dynamic_seq2seq/decoder/attention/multi_rnn_cell/cell_3/gru_cell/candidate/bias:0, (1000,), /device:GPU:0
  dynamic_seq2seq/decoder/attention/attention_layer/kernel:0, (2000, 1000), /device:GPU:0
  dynamic_seq2seq/decoder/output_projection/kernel:0, (1000, 2891), 
# log_file=../test_out/log_1552481826
  created train model with fresh parameters, time 1.24s
  created infer model with fresh parameters, time 0.21s
  # 301
    src: /features/fullFrame-227x227px/dev/20June_2011_Monday_heute-6514/
    ref: und eher wechselhaft geht es mit unserem wetter auch weiter .
    nmt: mäßig rum rum nordseeluft temperaturunterschied temperaturunterschied temperaturunterschied temperaturunterschied temperaturunterschied temperaturunterschied davon davon davon davon uhr uhr uhr uhr uhr frostig frostig frostig frostig frostig frostig frostig frostig frostig frostig frostig frostig frostig frostig frostig frostig frostig frostig neben neben neben
  created eval model with fresh parameters, time 0.15s
  eval dev: perplexity 4736.94, time 220s, Wed Mar 13 13:01:14 2019.
  eval test: perplexity 4779.63, time 270s, Wed Mar 13 13:05:45 2019.
  created infer model with fresh parameters, time 0.16s
# Start step 0, lr 1e-05, Wed Mar 13 13:05:45 2019
# Init train iterator, skipping 0 elements
  global step 100 lr 1e-05 step-time 1.35s wps 0.01K ppl 965.24 bleu 0.00
jackson1895 commented 5 years ago

hi , after using your solution, the same problem still occurs. I used a virtual environment with python 2.7.16 and tensorflow 1.4.0. As cuda I have 8.0 and cudnn 6.0.

mudit9 commented 4 years ago

Any updates on this?

nian-liu commented 4 years ago

Hi, I meet the same issue. Any update?

hshreeshail commented 1 year ago

I am facing the same issue. @jackson1895 @vbelissen did you find a solution?