Open allen20200111 opened 4 years ago
UnknownError Traceback (most recent call last) /home/xxx/tensorflow/lib/python3.6/site-packages/tensorflow/python/client/session.py in _do_call(self, fn, args) 1355 try: -> 1356 return fn(args) 1357 except errors.OpError as e:
/home/xxx/tensorflow/lib/python3.6/site-packages/tensorflow/python/client/session.py in _run_fn(feed_dict, fetch_list, target_list, options, run_metadata) 1340 return self._call_tf_sessionrun( -> 1341 options, feed_dict, fetch_list, target_list, run_metadata) 1342
/home/xxx/tensorflow/lib/python3.6/site-packages/tensorflow/python/client/session.py in _call_tf_sessionrun(self, options, feed_dict, fetch_list, target_list, run_metadata) 1428 self._session, options, feed_dict, fetch_list, target_list, -> 1429 run_metadata) 1430
UnknownError: 2 root error(s) found. (0) Unknown: CUDNN_STATUS_EXECUTION_FAILED in tensorflow/stream_executor/cuda/cuda_dnn.cc(953): 'cudnnSetDropoutDescriptor( handle.get(), cudnn.handle(), dropout, state_memory.opaque(), state_memory.size(), seed)' [[{{node model/cudnn_gru/CudnnRNN}}]] [[ConstantFoldingCtrl/model/absolute_difference/assert_broadcastable/AssertGuard/Switch_0/_30]] (1) Unknown: CUDNN_STATUS_EXECUTION_FAILED in tensorflow/stream_executor/cuda/cuda_dnn.cc(953): 'cudnnSetDropoutDescriptor( handle.get(), cudnn.handle(), dropout, state_memory.opaque(), state_memory.size(), seed)' [[{{node model/cudnn_gru/CudnnRNN}}]] 0 successful operations. 0 derived errors ignored.
During handling of the above exception, another exception occurred:
UnknownError Traceback (most recent call last) /home/xxx/tensorflow/lib/python3.6/site-packages/tensorflow/python/client/session.py in _do_call(self, fn, args) 1355 try: -> 1356 return fn(args) 1357 except errors.OpError as e:
/home/xxx/tensorflow/lib/python3.6/site-packages/tensorflow/python/client/session.py in _run_fn(feed_dict, fetch_list, target_list, options, run_metadata) 1340 return self._call_tf_sessionrun( -> 1341 options, feed_dict, fetch_list, target_list, run_metadata) 1342
/home/xxx/tensorflow/lib/python3.6/site-packages/tensorflow/python/client/session.py in _call_tf_sessionrun(self, options, feed_dict, fetch_list, target_list, run_metadata) 1428 self._session, options, feed_dict, fetch_list, target_list, -> 1429 run_metadata) 1430
UnknownError: 2 root error(s) found. (0) Unknown: CUDNN_STATUS_EXECUTION_FAILED in tensorflow/stream_executor/cuda/cuda_dnn.cc(953): 'cudnnSetDropoutDescriptor( handle.get(), cudnn.handle(), dropout, state_memory.opaque(), state_memory.size(), seed)' [[{{node model/cudnn_gru/CudnnRNN}}]] [[ConstantFoldingCtrl/model/absolute_difference/assert_broadcastable/AssertGuard/Switch_0/_30]] (1) Unknown: CUDNN_STATUS_EXECUTION_FAILED in tensorflow/stream_executor/cuda/cuda_dnn.cc(953): 'cudnnSetDropoutDescriptor( handle.get(), cudnn.handle(), dropout, state_memory.opaque(), state_memory.size(), seed)' [[{{node model/cudnn_gru/CudnnRNN}}]] 0 successful operations. 0 derived errors ignored.
During handling of the above exception, another exception occurred: