jk78346 commented 6 years ago

tf version: 1.1.0 python version: 3.4.5

2018-03-15T11:38:40.251583: step 1, loss 68.8194 Traceback (most recent call last): File "/dhome/jayhsu/miniconda2/envs/ENV/lib/python3.4/site-packages/tensorflow/python/client/session.py", line 1039, in _do_call return fn(*args) File "/dhome/jayhsu/miniconda2/envs/ENV/lib/python3.4/site-packages/tensorflow/python/client/session.py", line 1021, in _run_fn status, run_metadata) File "/dhome/jayhsu/miniconda2/envs/ENV/lib/python3.4/contextlib.py", line 66, in exit next(self.gen) File "/dhome/jayhsu/miniconda2/envs/ENV/lib/python3.4/site-packages/tensorflow/python/framework/errors_impl.py", line 466, in raise_exception_on_not_ok_status pywrap_tensorflow.TF_GetCode(status)) tensorflow.python.framework.errors_impl.InvalidArgumentError: Nan in summary histogram for: transitions_0/grad/hist [[Node: transitions_0/grad/hist = HistogramSummary[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"](transitions_0/grad/hist/tag, gradients/AddN_4)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "BasicTextPreprocessing_CNN_CRF.py", line 298, in embedd_table,char_batch,char_embedd_table) File "BasicTextPreprocessing_CNN_CRF.py", line 284, in train_step feed_dict) File "/dhome/jayhsu/miniconda2/envs/ENV/lib/python3.4/site-packages/tensorflow/python/client/session.py", line 778, in run run_metadata_ptr) File "/dhome/jayhsu/miniconda2/envs/ENV/lib/python3.4/site-packages/tensorflow/python/client/session.py", line 982, in _run feed_dict_string, options, run_metadata) File "/dhome/jayhsu/miniconda2/envs/ENV/lib/python3.4/site-packages/tensorflow/python/client/session.py", line 1032, in _do_run target_list, options, run_metadata) File "/dhome/jayhsu/miniconda2/envs/ENV/lib/python3.4/site-packages/tensorflow/python/client/session.py", line 1052, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.InvalidArgumentError: Nan in summary histogram for: transitions_0/grad/hist [[Node: transitions_0/grad/hist = HistogramSummary[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"](transitions_0/grad/hist/tag, gradients/AddN_4)]]

Caused by op 'transitions_0/grad/hist', defined at: File "BasicTextPreprocessing_CNN_CRF.py", line 226, in grad_hist_summary = tf.summary.histogram("{}/grad/hist".format(v.name), g) File "/dhome/jayhsu/miniconda2/envs/ENV/lib/python3.4/site-packages/tensorflow/python/summary/summary.py", line 209, in histogram tag=scope.rstrip('/'), values=values, name=scope) File "/dhome/jayhsu/miniconda2/envs/ENV/lib/python3.4/site-packages/tensorflow/python/ops/gen_logging_ops.py", line 139, in _histogram_summary name=name) File "/dhome/jayhsu/miniconda2/envs/ENV/lib/python3.4/site-packages/tensorflow/python/framework/op_def_library.py", line 768, in apply_op op_def=op_def) File "/dhome/jayhsu/miniconda2/envs/ENV/lib/python3.4/site-packages/tensorflow/python/framework/ops.py", line 2336, in create_op original_op=self._default_original_op, op_def=op_def) File "/dhome/jayhsu/miniconda2/envs/ENV/lib/python3.4/site-packages/tensorflow/python/framework/ops.py", line 1228, in init self._traceback = _extract_stack()

InvalidArgumentError (see above for traceback): Nan in summary histogram for: transitions_0/grad/hist [[Node: transitions_0/grad/hist = HistogramSummary[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"](transitions_0/grad/hist/tag, gradients/AddN_4)]]

Exception ignored in: <bound method Session.del of <tensorflow.python.client.session.Session object at 0x7ff256b115f8>> Traceback (most recent call last): File "/dhome/jayhsu/miniconda2/envs/ENV/lib/python3.4/site-packages/tensorflow/python/client/session.py", line 587, in del AttributeError: 'NoneType' object has no attribute 'TF_NewStatus'