THUNLP-MT / THUMT

An open-source neural machine translation toolkit developed by Tsinghua Natural Language Processing Group
BSD 3-Clause "New" or "Revised" License
703 stars 197 forks source link

RuntimeError:Graph is finalized and cannot be modified. when validating #46

Closed jingyihiter closed 6 years ago

jingyihiter commented 6 years ago

环境: python 2.7.15 tensorflow 1.7.0 运行命令: python bin/trainer.py --input mt_dataset/corpus.tc.32k.de.shuf mt_dataset/corpus.tc.32k.en.shuf --vocabulary vocab.32k.de.txt vocab.32k.en.txt --model transformer --validation mt_dataset/newstest2014.tc.32k.de --references mt_dataset/newstest2014.tc.32k.en --parameters=batch_size=6250,device_list=[3],train_steps=2000,eval_steps=100 在validating时出现如下错误:

Traceback (most recent call last):
  File "bin/trainer.py", line 472, in <module>
    main(parse_args())
  File "bin/trainer.py", line 468, in main
            "source_length": tf.placeholder(tf.int32, [None], "source_length")
    sess.run_step_fn(step_fn)
  File "/data1/xxx/anaconda3/envs/python2/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 607, in run_step_fn
    return self._sess.run_step_fn(step_fn, self._tf_sess(), run_with_hooks=None)
  File "/data1/xxx/anaconda3/envs/python2/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 1038, in run_step_fn
    return self._sess.run_step_fn(step_fn, raw_session, run_with_hooks)
  File "/data1/xxx/anaconda3/envs/python2/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 957, in run_step_fn
    return step_fn(_MonitoredSession.StepContext(raw_session, run_with_hooks))
  File "bin/trainer.py", line 458, in step_fn
    return step_context.run_with_hooks(ops["train_op"])
  File "/data1/xxx/anaconda3/envs/python2/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 634, in run_with_hooks
    return self._run_with_hooks_fn(*args, **kwargs)
  File "/data1/xxx/anaconda3/envs/python2/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 1113, in run
    raise six.reraise(*original_exc_info)
  File "/data1/xxx/anaconda3/envs/python2/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 1098, in run
    return self._sess.run(*args, **kwargs)
  File "/data1/xxx/anaconda3/envs/python2/lib/python2.7/site-packages/tensorflow/python/training/monitored_session.py", line 1178, in run
    run_metadata=run_metadata))
  File "/data1/xxx/gen_comment/THUMT/thumt/utils/hooks.py", line 279, in after_run
    self._session_config)
  File "/data1/xxx/gen_comment/THUMT/thumt/utils/hooks.py", line 168, in _evaluate
    print(tf.shape(outputs))
  File "/data1/xxx/anaconda3/envs/python2/lib/python2.7/site-packages/tensorflow/python/ops/array_ops.py", line 285, in shape
    return shape_internal(input, name, optimize=True, out_type=out_type)
  File "/data1/xxx/anaconda3/envs/python2/lib/python2.7/site-packages/tensorflow/python/ops/array_ops.py", line 309, in shape_internal
    input_tensor = ops.convert_to_tensor(input)
  File "/data1/xxx/anaconda3/envs/python2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 950, in convert_to_tensor
    as_ref=False)
  File "/data1/xxx/anaconda3/envs/python2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1040, in internal_convert_to_tensor
    ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
  File "/data1/xxx/anaconda3/envs/python2/lib/python2.7/site-packages/tensorflow/python/framework/constant_op.py", line 235, in _constant_tensor_conversion_function
    return constant(v, dtype=dtype, name=name)
  File "/data1/xxx/anaconda3/envs/python2/lib/python2.7/site-packages/tensorflow/python/framework/constant_op.py", line 220, in constant
    name=name).outputs[0]
  File "/data1/xxx/anaconda3/envs/python2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 3262, in create_op
    self._check_not_finalized()
  File "/data1/xxx/anaconda3/envs/python2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2896, in _check_not_finalized
    raise RuntimeError("Graph is finalized and cannot be modified.")```

打印调试发现是在这个地方出错:
utils/hooks.py
```python
        with tf.train.MonitoredSession(session_creator=sess_creator) as sess:
            while not sess.should_stop():
                feats = sess.run(features)
                outputs = sess.run(predictions, feed_dict={
                    placeholders["source"]: feats["source"],
                    placeholders["source_length"]: feats["source_length"]
                })
                # shape: [batch, len]
                outputs = outputs.tolist()
                # shape: ([batch, len], ..., [batch, len])
jingyihiter commented 6 years ago

解决了这个问题,是我自己的问题。后面加错了一行打印的代码(: