BishnoiAmit
commented
6 years ago Issue has been temporarily resolved by switching to CPU in place of GPU. Permanent solution is yet to be discovered.
Open BishnoiAmit opened 6 years ago
BishnoiAmit
commented
6 years ago Issue has been temporarily resolved by switching to CPU in place of GPU. Permanent solution is yet to be discovered.
During the MultilayerPerceptron.py execution of the code, execution was interrupted. Problem could be buffer related, please try to resolve as soon as possible.
2018-08-28 19:45:00.440222: W T:\src\github\tensorflow\tensorflow\core\common_runtime\bfc_allocator.cc:279] **** 2018-08-28 19:45:00.445770: W T:\src\github\tensorflow\tensorflow\core\framework\op_kernel.cc:1275] OP_REQUIRES failed at matmul_op.cc:474 : Resource exhausted: OOM when allocating tensor with shape[1240,4000] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc Traceback (most recent call last): File "D:\Softwares_Installation\Anaconda_Installed\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1278, in _do_call return fn(*args) File "D:\Softwares_Installation\Anaconda_Installed\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1263, in _run_fn options, feed_dict, fetch_list, target_list, run_metadata) File "D:\Softwares_Installation\Anaconda_Installed\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1350, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[1240,4000] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[Node: MatMul = MatMul[T=DT_FLOAT, transpose_a=false, transpose_b=false, _device="/job:localhost/replica:0/task:0/device:GPU:0"](_arg_Placeholder_0_0/_3, Variable/read)]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "MultilayerPerceptron.py", line 194, in
main()
File "MultilayerPerceptron.py", line 180, in main
obj.train_model(prediction, X_train, X_test, Y_train, Y_test)
File "MultilayerPerceptron.py", line 99, in trainmodel
, c = sess.run([optimizer, cost], feed_dict={self.x:X_train, self.y:y_train})
File "D:\Softwares_Installation\Anaconda_Installed\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 877, in run
run_metadata_ptr)
File "D:\Softwares_Installation\Anaconda_Installed\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1100, in _run
feed_dict_tensor, options, run_metadata)
File "D:\Softwares_Installation\Anaconda_Installed\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1272, in _do_run
run_metadata)
File "D:\Softwares_Installation\Anaconda_Installed\envs\tensorflow\lib\site-packages\tensorflow\python\client\session.py", line 1291, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[1240,4000] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[Node: MatMul = MatMul[T=DT_FLOAT, transpose_a=false, transpose_b=false, _device="/job:localhost/replica:0/task:0/device:GPU:0"](_arg_Placeholder_0_0/_3, Variable/read)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
Caused by op 'MatMul', defined at: File "MultilayerPerceptron.py", line 194, in
main()
File "MultilayerPerceptron.py", line 130, in main
prediction = obj.ModelGraph()
File "MultilayerPerceptron.py", line 51, in ModelGraph
layer_1 = tf.add(tf.matmul(self.x, self.weights['h1']), self.biases['b1'])
File "D:\Softwares_Installation\Anaconda_Installed\envs\tensorflow\lib\site-packages\tensorflow\python\ops\math_ops.py", line 2018, in matmul
a, b, transpose_a=transpose_a, transpose_b=transpose_b, name=name)
File "D:\Softwares_Installation\Anaconda_Installed\envs\tensorflow\lib\site-packages\tensorflow\python\ops\gen_math_ops.py", line 4750, in mat_mul
name=name)
File "D:\Softwares_Installation\Anaconda_Installed\envs\tensorflow\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "D:\Softwares_Installation\Anaconda_Installed\envs\tensorflow\lib\site-packages\tensorflow\python\util\deprecation.py", line 454, in new_func
return func(*args, **kwargs)
File "D:\Softwares_Installation\Anaconda_Installed\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 3155, in create_op
op_def=op_def)
File "D:\Softwares_Installation\Anaconda_Installed\envs\tensorflow\lib\site-packages\tensorflow\python\framework\ops.py", line 1717, in init
self._traceback = tf_stack.extract_stack()
ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[1240,4000] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[Node: MatMul = MatMul[T=DT_FLOAT, transpose_a=false, transpose_b=false, _device="/job:localhost/replica:0/task:0/device:GPU:0"](_arg_Placeholder_0_0/_3, Variable/read)]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info. @HarikrishnaRanpariya #3