`
2019-10-24 21:55:25.500656: W tensorflow/core/common_runtime/bfc_allocator.cc:279] ****
2019-10-24 21:55:25.500952: W tensorflow/core/framework/op_kernel.cc:1318] OP_REQUIRES failed at cwise_ops_common.cc:70 : Resource exhausted: OOM when allocating tensor with shape[8,509,3072] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
Traceback (most recent call last):
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1322, in _do_call
return fn(*args)
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1307, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1409, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[8,509,3072] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[Node: gradients/model/transformer/layer_20/ff/drop_1/dropout/mul_grad/Mul = Mul[T=DT_FLOAT, _grappler:ArithmeticOptimizer:MinimizeBroadcasts=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](ConstantFolding/gradients/model/transformer/dropout/dropout/div_grad/RealDiv_recip, model/transformer/layer_20/ff/drop_1/dropout/Floor)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "model.py", line 429, in
tf.app.run()
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))
File "model.py", line 424, in main
model.train()
File "model.py", line 329, in train
global_steps, loss, logits, acc, length = self.xlnet_step(sess, batch)
File "model.py", line 270, in xlnet_step
embedding, globalsteps, loss, , logits, acc, length = sess.run([self.embedded, self.global_steps, self.loss, self.train_op, self.logits, self.accuracy, self.length], feed_dict=feed)
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 900, in run
run_metadata_ptr)
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1135, in _run
feed_dict_tensor, options, run_metadata)
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1316, in _do_run
run_metadata)
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1335, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[8,509,3072] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[Node: gradients/model/transformer/layer_20/ff/drop_1/dropout/mul_grad/Mul = Mul[T=DT_FLOAT, _grappler:ArithmeticOptimizer:MinimizeBroadcasts=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](ConstantFolding/gradients/model/transformer/dropout/dropout/div_grad/RealDiv_recip, model/transformer/layer_20/ff/drop_1/dropout/Floor)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
Caused by op 'gradients/model/transformer/layer_20/ff/drop_1/dropout/mul_grad/Mul', defined at:
File "model.py", line 429, in
tf.app.run()
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))
File "model.py", line 424, in main
model.train()
File "model.py", line 302, in train
self.creat_model()
File "model.py", line 110, in __creat_model
self.xlnet_optimizer_layer()
File "model.py", line 253, in xlnet_optimizer_layer
self.train_op, self.learningrate, = model_utils.get_train_op(FLAGS, self.loss)
File "/home/eric/Documents/projects/XLNet-ChineseNER/model_utils.py", line 145, in get_train_op
grads_and_vars = optimizer.compute_gradients(total_loss)
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/optimizer.py", line 511, in compute_gradients
colocate_gradients_with_ops=colocate_gradients_with_ops)
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py", line 532, in gradients
gate_gradients, aggregation_method, stop_gradients)
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py", line 701, in _GradientsHelper
lambda: grad_fn(op, out_grads))
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py", line 396, in _MaybeCompile
return grad_fn() # Exit early
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py", line 701, in
lambda: grad_fn(op, out_grads))
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/math_grad.py", line 880, in _MulGrad
math_ops.reduce_sum(gen_math_ops.mul(grad, y), rx), sx),
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/gen_math_ops.py", line 4759, in mul
"Mul", x=x, y=y, name=name)
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3414, in create_op
op_def=op_def)
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1740, in init__
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access
...which was originally created as op 'model/transformer/layer_20/ff/drop_1/dropout/mul', defined at:
File "model.py", line 429, in
tf.app.run()
[elided 2 identical lines from previous traceback]
File "model.py", line 302, in train
self.creat_model()
File "model.py", line 95, in __creat_model
self.xlnet_layer()
File "model.py", line 153, in xlnet_layer
input_mask = self.input_mask)
File "/home/eric/Documents/projects/XLNet-ChineseNER/xlnet.py", line 222, in init
) = modeling.transformer_xl(**tfm_args)
File "/home/eric/Documents/projects/XLNet-ChineseNER/modeling.py", line 649, in transformer_xl
reuse=reuse)
File "/home/eric/Documents/projects/XLNet-ChineseNER/modeling.py", line 69, in positionwise_ffn
name='drop_1')
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/layers/core.py", line 271, in dropout
return layer.apply(inputs, training=training)
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 774, in apply
return self.call(inputs, *args, **kwargs)
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/layers/base.py", line 329, in call
outputs = super(Layer, self).call__(inputs, *args, *kwargs)
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 703, in call
outputs = self.call(inputs, args, **kwargs)
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/layers/core.py", line 229, in call
return super(Dropout, self).call(inputs, training=training)
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/keras/layers/core.py", line 144, in call
lambda: array_ops.identity(inputs))
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/keras/utils/tf_utils.py", line 51, in smart_cond
pred, true_fn=true_fn, false_fn=false_fn, name=name)
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/smart_cond.py", line 54, in smart_cond
return true_fn()
ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[8,509,3072] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[Node: gradients/model/transformer/layer_20/ff/drop_1/dropout/mul_grad/Mul = Mul[T=DT_FLOAT, _grappler:ArithmeticOptimizer:MinimizeBroadcasts=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](ConstantFolding/gradients/model/transformer/dropout/dropout/div_grad/RealDiv_recip, model/transformer/layer_20/ff/drop_1/dropout/Floor)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
w5688414
commented
5 years ago
` 2019-10-24 21:55:25.500656: W tensorflow/core/common_runtime/bfc_allocator.cc:279] **** 2019-10-24 21:55:25.500952: W tensorflow/core/framework/op_kernel.cc:1318] OP_REQUIRES failed at cwise_ops_common.cc:70 : Resource exhausted: OOM when allocating tensor with shape[8,509,3072] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc Traceback (most recent call last): File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1322, in _do_call return fn(*args) File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1307, in _run_fn options, feed_dict, fetch_list, target_list, run_metadata) File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1409, in _call_tf_sessionrun run_metadata) tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[8,509,3072] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[Node: gradients/model/transformer/layer_20/ff/drop_1/dropout/mul_grad/Mul = Mul[T=DT_FLOAT, _grappler:ArithmeticOptimizer:MinimizeBroadcasts=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](ConstantFolding/gradients/model/transformer/dropout/dropout/div_grad/RealDiv_recip, model/transformer/layer_20/ff/drop_1/dropout/Floor)]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "model.py", line 429, in
tf.app.run()
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))
File "model.py", line 424, in main
model.train()
File "model.py", line 329, in train
global_steps, loss, logits, acc, length = self.xlnet_step(sess, batch)
File "model.py", line 270, in xlnet_step
embedding, globalsteps, loss, , logits, acc, length = sess.run([self.embedded, self.global_steps, self.loss, self.train_op, self.logits, self.accuracy, self.length], feed_dict=feed)
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 900, in run
run_metadata_ptr)
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1135, in _run
feed_dict_tensor, options, run_metadata)
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1316, in _do_run
run_metadata)
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1335, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[8,509,3072] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[Node: gradients/model/transformer/layer_20/ff/drop_1/dropout/mul_grad/Mul = Mul[T=DT_FLOAT, _grappler:ArithmeticOptimizer:MinimizeBroadcasts=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](ConstantFolding/gradients/model/transformer/dropout/dropout/div_grad/RealDiv_recip, model/transformer/layer_20/ff/drop_1/dropout/Floor)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
Caused by op 'gradients/model/transformer/layer_20/ff/drop_1/dropout/mul_grad/Mul', defined at: File "model.py", line 429, in
tf.app.run()
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/platform/app.py", line 125, in run
_sys.exit(main(argv))
File "model.py", line 424, in main
model.train()
File "model.py", line 302, in train
self.creat_model()
File "model.py", line 110, in __creat_model
self.xlnet_optimizer_layer()
File "model.py", line 253, in xlnet_optimizer_layer
self.train_op, self.learningrate, = model_utils.get_train_op(FLAGS, self.loss)
File "/home/eric/Documents/projects/XLNet-ChineseNER/model_utils.py", line 145, in get_train_op
grads_and_vars = optimizer.compute_gradients(total_loss)
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/optimizer.py", line 511, in compute_gradients
colocate_gradients_with_ops=colocate_gradients_with_ops)
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py", line 532, in gradients
gate_gradients, aggregation_method, stop_gradients)
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py", line 701, in _GradientsHelper
lambda: grad_fn(op, out_grads))
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py", line 396, in _MaybeCompile
return grad_fn() # Exit early
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py", line 701, in
lambda: grad_fn(op, out_grads))
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/math_grad.py", line 880, in _MulGrad
math_ops.reduce_sum(gen_math_ops.mul(grad, y), rx), sx),
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/ops/gen_math_ops.py", line 4759, in mul
"Mul", x=x, y=y, name=name)
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3414, in create_op
op_def=op_def)
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1740, in init__
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access
...which was originally created as op 'model/transformer/layer_20/ff/drop_1/dropout/mul', defined at: File "model.py", line 429, in
tf.app.run()
[elided 2 identical lines from previous traceback]
File "model.py", line 302, in train
self.creat_model()
File "model.py", line 95, in __creat_model
self.xlnet_layer()
File "model.py", line 153, in xlnet_layer
input_mask = self.input_mask)
File "/home/eric/Documents/projects/XLNet-ChineseNER/xlnet.py", line 222, in init
) = modeling.transformer_xl(**tfm_args)
File "/home/eric/Documents/projects/XLNet-ChineseNER/modeling.py", line 649, in transformer_xl
reuse=reuse)
File "/home/eric/Documents/projects/XLNet-ChineseNER/modeling.py", line 69, in positionwise_ffn
name='drop_1')
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/layers/core.py", line 271, in dropout
return layer.apply(inputs, training=training)
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 774, in apply
return self.call(inputs, *args, **kwargs)
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/layers/base.py", line 329, in call
outputs = super(Layer, self).call__(inputs, *args, *kwargs)
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/keras/engine/base_layer.py", line 703, in call
outputs = self.call(inputs, args, **kwargs)
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/layers/core.py", line 229, in call
return super(Dropout, self).call(inputs, training=training)
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/keras/layers/core.py", line 144, in call
lambda: array_ops.identity(inputs))
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/keras/utils/tf_utils.py", line 51, in smart_cond
pred, true_fn=true_fn, false_fn=false_fn, name=name)
File "/home/eric/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/smart_cond.py", line 54, in smart_cond
return true_fn()
ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[8,509,3072] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc [[Node: gradients/model/transformer/layer_20/ff/drop_1/dropout/mul_grad/Mul = Mul[T=DT_FLOAT, _grappler:ArithmeticOptimizer:MinimizeBroadcasts=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](ConstantFolding/gradients/model/transformer/dropout/dropout/div_grad/RealDiv_recip, model/transformer/layer_20/ff/drop_1/dropout/Floor)]] Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
`