My Environment:
win10
python 3.6
tensorflow 1.4
gpu gtx965m 2g
nvcc -V 8.0
cudnn 6.0
It took me a whole day didn't solve the problem 0.0
ps:
Please switch to tf.train.create_global_step
INFO:tensorflow:Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
WARNING:tensorflow:From E:\TensorFlow\TensorFlow Object Detection API Tutorial\models\research\object_detection\meta_architectures\faster_rcnn_meta_arch.py:1670: get_or_create_global_step (from tensorflow.contrib.framework.python.ops.variables) is deprecated and will be removed in a future version.
Instructions for updating:
Please switch to tf.train.get_or_create_global_step
INFO:tensorflow:Summary name /clone_loss is illegal; using clone_loss instead.
D:\python36\lib\site-packages\tensorflow\python\ops\gradients_impl.py:96: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
"Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
INFO:tensorflow:Restoring parameters from trainmodels/model.ckpt
INFO:tensorflow:Starting Session.
INFO:tensorflow:Saving checkpoint to path train/model.ckpt
INFO:tensorflow:Starting Queues.
INFO:tensorflow:global_step/sec: 0
INFO:tensorflow:Error reported to Coordinator: <class 'tensorflow.python.framework.errors_impl.ResourceExhaustedError'>, OOM when allocating tensor with shape[4,38,56,1088]
[[Node: FirstStageFeatureExtractor/InceptionResnetV2/InceptionResnetV2/Repeat_1/block17_3/Conv2d_1x1/Conv2D = Conv2D[T=DT_FLOAT, data_format="NHWC", padding="VALID", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](FirstStageFeatureExtractor/InceptionResnetV2/InceptionResnetV2/Repeat_1/block17_3/Conv2d_1x1/SpaceToBatchND, FirstStageFeatureExtractor/InceptionResnetV2/Repeat_1/block17_3/Conv2d_1x1/weights/read/_945)]]
[[Node: BatchMultiClassNonMaxSuppression/map/while/Switch_5/_1739 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_10664_BatchMultiClassNonMaxSuppression/map/while/Switch_5", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
Caused by op 'FirstStageFeatureExtractor/InceptionResnetV2/InceptionResnetV2/Repeat_1/block17_3/Conv2d_1x1/Conv2D', defined at:
File "train.py", line 167, in
tf.app.run()
File "D:\python36\lib\site-packages\tensorflow\python\platform\app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "train.py", line 163, in main
worker_job_name, is_chief, FLAGS.train_dir)
File "E:\TensorFlow\TensorFlow Object Detection API Tutorial\models\research\object_detection\trainer.py", line 228, in train
clones = model_deploy.create_clones(deploy_config, model_fn, [input_queue])
File "E:\TensorFlow\TensorFlow Object Detection API Tutorial\models\research\slim\deployment\model_deploy.py", line 193, in create_clones
outputs = model_fn(*args, kwargs)
File "E:\TensorFlow\TensorFlow Object Detection API Tutorial\models\research\object_detection\trainer.py", line 165, in _create_losses
prediction_dict = detection_model.predict(images)
File "E:\TensorFlow\TensorFlow Object Detection API Tutorial\models\research\object_detection\meta_architectures\faster_rcnn_meta_arch.py", line 531, in predict
image_shape) = self._extract_rpn_feature_maps(preprocessed_inputs)
File "E:\TensorFlow\TensorFlow Object Detection API Tutorial\models\research\object_detection\meta_architectures\faster_rcnn_meta_arch.py", line 685, in _extract_rpn_feature_maps
preprocessed_inputs, scope=self.first_stage_feature_extractor_scope)
File "E:\TensorFlow\TensorFlow Object Detection API Tutorial\models\research\object_detection\meta_architectures\faster_rcnn_meta_arch.py", line 134, in extract_proposal_features
return self._extract_proposal_features(preprocessed_inputs, scope)
File "E:\TensorFlow\TensorFlow Object Detection API Tutorial\models\research\object_detection\models\faster_rcnn_inception_resnet_v2_feature_extractor.py", line 112, in _extract_proposal_features
align_feature_maps=True))
File "E:\TensorFlow\TensorFlow Object Detection API Tutorial\models\research\slim\nets\inception_resnet_v2.py", line 244, in inception_resnet_v2_base
activation_fn=activation_fn)
File "D:\python36\lib\site-packages\tensorflow\contrib\layers\python\layers\layers.py", line 2346, in repeat
outputs = layer(outputs, *args, *kwargs)
File "E:\TensorFlow\TensorFlow Object Detection API Tutorial\models\research\slim\nets\inception_resnet_v2.py", line 72, in block17
activation_fn=None, scope='Conv2d_1x1')
File "D:\python36\lib\site-packages\tensorflow\contrib\framework\python\ops\arg_scope.py", line 181, in func_with_args
return func(args, current_args)
File "D:\python36\lib\site-packages\tensorflow\contrib\layers\python\layers\layers.py", line 1033, in convolution
outputs = layer.apply(inputs)
File "D:\python36\lib\site-packages\tensorflow\python\layers\base.py", line 671, in apply
return self.call(inputs, *args, *kwargs)
File "D:\python36\lib\site-packages\tensorflow\python\layers\base.py", line 575, in call
outputs = self.call(inputs, args, **kwargs)
File "D:\python36\lib\site-packages\tensorflow\python\layers\convolutional.py", line 167, in call
outputs = self._convolution_op(inputs, self.kernel)
File "D:\python36\lib\site-packages\tensorflow\python\ops\nn_ops.py", line 835, in call
return self.conv_op(inp, filter)
File "D:\python36\lib\site-packages\tensorflow\python\ops\nn_ops.py", line 499, in call
return self.call(inp, filter)
File "D:\python36\lib\site-packages\tensorflow\python\ops\nn_ops.py", line 492, in _with_space_to_batch_call
result = self.op(input_converted, filter)
File "D:\python36\lib\site-packages\tensorflow\python\ops\nn_ops.py", line 187, in call
name=self.name)
File "D:\python36\lib\site-packages\tensorflow\python\ops\gen_nn_ops.py", line 630, in conv2d
data_format=data_format, name=name)
File "D:\python36\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "D:\python36\lib\site-packages\tensorflow\python\framework\ops.py", line 2956, in create_op
op_def=op_def)
File "D:\python36\lib\site-packages\tensorflow\python\framework\ops.py", line 1470, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access
Traceback (most recent call last):
File "D:\python36\lib\site-packages\tensorflow\python\client\session.py", line 1323, in _do_call
return fn(*args)
File "D:\python36\lib\site-packages\tensorflow\python\client\session.py", line 1302, in _run_fn
status, run_metadata)
File "D:\python36\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 473, in exit
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[1,75,100,1088]
[[Node: FirstStageFeatureExtractor/InceptionResnetV2/InceptionResnetV2/Repeat_1/block17_1/Conv2d_1x1/BatchToSpaceND = BatchToSpaceND[T=DT_FLOAT, Tblock_shape=DT_INT32, Tcrops=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"](FirstStageFeatureExtractor/InceptionResnetV2/InceptionResnetV2/Repeat_1/block17_1/Conv2d_1x1/Conv2D, Conv/SpaceToBatchND/block_shape, FirstStageFeatureExtractor/InceptionResnetV2/InceptionResnetV2/Repeat_1/block17_1/Conv2d_1x1/strided_slice_3)]]
[[Node: gradients/SecondStageFeatureExtractor/InceptionResnetV2/Block8/Branch_1/Conv2d_0a_1x1/Conv2D_grad/tuple/control_dependency_1/_4565 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_13255_gradients/SecondStageFeatureExtractor/InceptionResnetV2/Block8/Branch_1/Conv2d_0a_1x1/Conv2D_grad/tuple/control_dependency_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "D:\python36\lib\site-packages\tensorflow\python\training\supervisor.py", line 954, in managed_session
yield sess
File "D:\python36\lib\site-packages\tensorflow\contrib\slim\python\slim\learning.py", line 763, in train
sess, train_op, global_step, train_step_kwargs)
File "D:\python36\lib\site-packages\tensorflow\contrib\slim\python\slim\learning.py", line 487, in train_step
run_metadata=run_metadata)
File "D:\python36\lib\site-packages\tensorflow\python\client\session.py", line 889, in run
run_metadata_ptr)
File "D:\python36\lib\site-packages\tensorflow\python\client\session.py", line 1120, in _run
feed_dict_tensor, options, run_metadata)
File "D:\python36\lib\site-packages\tensorflow\python\client\session.py", line 1317, in _do_run
options, run_metadata)
File "D:\python36\lib\site-packages\tensorflow\python\client\session.py", line 1336, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[1,75,100,1088]
[[Node: FirstStageFeatureExtractor/InceptionResnetV2/InceptionResnetV2/Repeat_1/block17_1/Conv2d_1x1/BatchToSpaceND = BatchToSpaceND[T=DT_FLOAT, Tblock_shape=DT_INT32, Tcrops=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"](FirstStageFeatureExtractor/InceptionResnetV2/InceptionResnetV2/Repeat_1/block17_1/Conv2d_1x1/Conv2D, Conv/SpaceToBatchND/block_shape, FirstStageFeatureExtractor/InceptionResnetV2/InceptionResnetV2/Repeat_1/block17_1/Conv2d_1x1/strided_slice_3)]]
[[Node: gradients/SecondStageFeatureExtractor/InceptionResnetV2/Block8/Branch_1/Conv2d_0a_1x1/Conv2D_grad/tuple/control_dependency_1/_4565 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_13255_gradients/SecondStageFeatureExtractor/InceptionResnetV2/Block8/Branch_1/Conv2d_0a_1x1/Conv2D_grad/tuple/control_dependency_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Caused by op 'FirstStageFeatureExtractor/InceptionResnetV2/InceptionResnetV2/Repeat_1/block17_1/Conv2d_1x1/BatchToSpaceND', defined at:
File "train.py", line 167, in
tf.app.run()
File "D:\python36\lib\site-packages\tensorflow\python\platform\app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "train.py", line 163, in main
worker_job_name, is_chief, FLAGS.train_dir)
File "E:\TensorFlow\TensorFlow Object Detection API Tutorial\models\research\object_detection\trainer.py", line 228, in train
clones = model_deploy.create_clones(deploy_config, model_fn, [input_queue])
File "E:\TensorFlow\TensorFlow Object Detection API Tutorial\models\research\slim\deployment\model_deploy.py", line 193, in create_clones
outputs = model_fn(*args, kwargs)
File "E:\TensorFlow\TensorFlow Object Detection API Tutorial\models\research\object_detection\trainer.py", line 165, in _create_losses
prediction_dict = detection_model.predict(images)
File "E:\TensorFlow\TensorFlow Object Detection API Tutorial\models\research\object_detection\meta_architectures\faster_rcnn_meta_arch.py", line 531, in predict
image_shape) = self._extract_rpn_feature_maps(preprocessed_inputs)
File "E:\TensorFlow\TensorFlow Object Detection API Tutorial\models\research\object_detection\meta_architectures\faster_rcnn_meta_arch.py", line 685, in _extract_rpn_feature_maps
preprocessed_inputs, scope=self.first_stage_feature_extractor_scope)
File "E:\TensorFlow\TensorFlow Object Detection API Tutorial\models\research\object_detection\meta_architectures\faster_rcnn_meta_arch.py", line 134, in extract_proposal_features
return self._extract_proposal_features(preprocessed_inputs, scope)
File "E:\TensorFlow\TensorFlow Object Detection API Tutorial\models\research\object_detection\models\faster_rcnn_inception_resnet_v2_feature_extractor.py", line 112, in _extract_proposal_features
align_feature_maps=True))
File "E:\TensorFlow\TensorFlow Object Detection API Tutorial\models\research\slim\nets\inception_resnet_v2.py", line 244, in inception_resnet_v2_base
activation_fn=activation_fn)
File "D:\python36\lib\site-packages\tensorflow\contrib\layers\python\layers\layers.py", line 2346, in repeat
outputs = layer(outputs, *args, *kwargs)
File "E:\TensorFlow\TensorFlow Object Detection API Tutorial\models\research\slim\nets\inception_resnet_v2.py", line 72, in block17
activation_fn=None, scope='Conv2d_1x1')
File "D:\python36\lib\site-packages\tensorflow\contrib\framework\python\ops\arg_scope.py", line 181, in func_with_args
return func(args, current_args)
File "D:\python36\lib\site-packages\tensorflow\contrib\layers\python\layers\layers.py", line 1033, in convolution
outputs = layer.apply(inputs)
File "D:\python36\lib\site-packages\tensorflow\python\layers\base.py", line 671, in apply
return self.call(inputs, *args, *kwargs)
File "D:\python36\lib\site-packages\tensorflow\python\layers\base.py", line 575, in call
outputs = self.call(inputs, args, **kwargs)
File "D:\python36\lib\site-packages\tensorflow\python\layers\convolutional.py", line 167, in call
outputs = self._convolution_op(inputs, self.kernel)
File "D:\python36\lib\site-packages\tensorflow\python\ops\nn_ops.py", line 835, in call
return self.conv_op(inp, filter)
File "D:\python36\lib\site-packages\tensorflow\python\ops\nn_ops.py", line 499, in call
return self.call(inp, filter)
File "D:\python36\lib\site-packages\tensorflow\python\ops\nn_ops.py", line 495, in _with_space_to_batch_call
input=result, block_shape=dilation_rate, crops=crops)
File "D:\python36\lib\site-packages\tensorflow\python\ops\gen_array_ops.py", line 404, in batch_to_space_nd
name=name)
File "D:\python36\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "D:\python36\lib\site-packages\tensorflow\python\framework\ops.py", line 2956, in create_op
op_def=op_def)
File "D:\python36\lib\site-packages\tensorflow\python\framework\ops.py", line 1470, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access
ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[1,75,100,1088]
[[Node: FirstStageFeatureExtractor/InceptionResnetV2/InceptionResnetV2/Repeat_1/block17_1/Conv2d_1x1/BatchToSpaceND = BatchToSpaceND[T=DT_FLOAT, Tblock_shape=DT_INT32, Tcrops=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"](FirstStageFeatureExtractor/InceptionResnetV2/InceptionResnetV2/Repeat_1/block17_1/Conv2d_1x1/Conv2D, Conv/SpaceToBatchND/block_shape, FirstStageFeatureExtractor/InceptionResnetV2/InceptionResnetV2/Repeat_1/block17_1/Conv2d_1x1/strided_slice_3)]]
[[Node: gradients/SecondStageFeatureExtractor/InceptionResnetV2/Block8/Branch_1/Conv2d_0a_1x1/Conv2D_grad/tuple/control_dependency_1/_4565 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_13255_gradients/SecondStageFeatureExtractor/InceptionResnetV2/Block8/Branch_1/Conv2d_0a_1x1/Conv2D_grad/tuple/control_dependency_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "train.py", line 167, in
tf.app.run()
File "D:\python36\lib\site-packages\tensorflow\python\platform\app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "train.py", line 163, in main
worker_job_name, is_chief, FLAGS.train_dir)
File "E:\TensorFlow\TensorFlow Object Detection API Tutorial\models\research\object_detection\trainer.py", line 337, in train
saver=saver)
File "D:\python36\lib\site-packages\tensorflow\contrib\slim\python\slim\learning.py", line 775, in train
sv.stop(threads, close_summary_writer=True)
File "D:\python36\lib\contextlib.py", line 99, in exit
self.gen.throw(type, value, traceback)
File "D:\python36\lib\site-packages\tensorflow\python\training\supervisor.py", line 964, in managed_session
self.stop(close_summary_writer=close_summary_writer)
File "D:\python36\lib\site-packages\tensorflow\python\training\supervisor.py", line 792, in stop
stop_grace_period_secs=self._stop_grace_secs)
File "D:\python36\lib\site-packages\tensorflow\python\training\coordinator.py", line 389, in join
six.reraise(*self._exc_info_to_raise)
File "D:\python36\lib\site-packages\six.py", line 693, in reraise
raise value
File "D:\python36\lib\site-packages\tensorflow\python\training\coordinator.py", line 296, in stop_on_exception
yield
File "D:\python36\lib\site-packages\tensorflow\python\training\coordinator.py", line 494, in run
self.run_loop()
File "D:\python36\lib\site-packages\tensorflow\python\training\supervisor.py", line 994, in run_loop
self._sv.global_step])
File "D:\python36\lib\site-packages\tensorflow\python\client\session.py", line 889, in run
run_metadata_ptr)
File "D:\python36\lib\site-packages\tensorflow\python\client\session.py", line 1120, in _run
feed_dict_tensor, options, run_metadata)
File "D:\python36\lib\site-packages\tensorflow\python\client\session.py", line 1317, in _do_run
options, run_metadata)
File "D:\python36\lib\site-packages\tensorflow\python\client\session.py", line 1336, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[4,38,56,1088]
[[Node: FirstStageFeatureExtractor/InceptionResnetV2/InceptionResnetV2/Repeat_1/block17_3/Conv2d_1x1/Conv2D = Conv2D[T=DT_FLOAT, data_format="NHWC", padding="VALID", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](FirstStageFeatureExtractor/InceptionResnetV2/InceptionResnetV2/Repeat_1/block17_3/Conv2d_1x1/SpaceToBatchND, FirstStageFeatureExtractor/InceptionResnetV2/Repeat_1/block17_3/Conv2d_1x1/weights/read/_945)]]
[[Node: BatchMultiClassNonMaxSuppression/map/while/Switch_5/_1739 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_10664_BatchMultiClassNonMaxSuppression/map/while/Switch_5", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
Caused by op 'FirstStageFeatureExtractor/InceptionResnetV2/InceptionResnetV2/Repeat_1/block17_3/Conv2d_1x1/Conv2D', defined at:
File "train.py", line 167, in
tf.app.run()
File "D:\python36\lib\site-packages\tensorflow\python\platform\app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "train.py", line 163, in main
worker_job_name, is_chief, FLAGS.train_dir)
File "E:\TensorFlow\TensorFlow Object Detection API Tutorial\models\research\object_detection\trainer.py", line 228, in train
clones = model_deploy.create_clones(deploy_config, model_fn, [input_queue])
File "E:\TensorFlow\TensorFlow Object Detection API Tutorial\models\research\slim\deployment\model_deploy.py", line 193, in create_clones
outputs = model_fn(*args, kwargs)
File "E:\TensorFlow\TensorFlow Object Detection API Tutorial\models\research\object_detection\trainer.py", line 165, in _create_losses
prediction_dict = detection_model.predict(images)
File "E:\TensorFlow\TensorFlow Object Detection API Tutorial\models\research\object_detection\meta_architectures\faster_rcnn_meta_arch.py", line 531, in predict
image_shape) = self._extract_rpn_feature_maps(preprocessed_inputs)
File "E:\TensorFlow\TensorFlow Object Detection API Tutorial\models\research\object_detection\meta_architectures\faster_rcnn_meta_arch.py", line 685, in _extract_rpn_feature_maps
preprocessed_inputs, scope=self.first_stage_feature_extractor_scope)
File "E:\TensorFlow\TensorFlow Object Detection API Tutorial\models\research\object_detection\meta_architectures\faster_rcnn_meta_arch.py", line 134, in extract_proposal_features
return self._extract_proposal_features(preprocessed_inputs, scope)
File "E:\TensorFlow\TensorFlow Object Detection API Tutorial\models\research\object_detection\models\faster_rcnn_inception_resnet_v2_feature_extractor.py", line 112, in _extract_proposal_features
align_feature_maps=True))
File "E:\TensorFlow\TensorFlow Object Detection API Tutorial\models\research\slim\nets\inception_resnet_v2.py", line 244, in inception_resnet_v2_base
activation_fn=activation_fn)
File "D:\python36\lib\site-packages\tensorflow\contrib\layers\python\layers\layers.py", line 2346, in repeat
outputs = layer(outputs, *args, *kwargs)
File "E:\TensorFlow\TensorFlow Object Detection API Tutorial\models\research\slim\nets\inception_resnet_v2.py", line 72, in block17
activation_fn=None, scope='Conv2d_1x1')
File "D:\python36\lib\site-packages\tensorflow\contrib\framework\python\ops\arg_scope.py", line 181, in func_with_args
return func(args, current_args)
File "D:\python36\lib\site-packages\tensorflow\contrib\layers\python\layers\layers.py", line 1033, in convolution
outputs = layer.apply(inputs)
File "D:\python36\lib\site-packages\tensorflow\python\layers\base.py", line 671, in apply
return self.call(inputs, *args, *kwargs)
File "D:\python36\lib\site-packages\tensorflow\python\layers\base.py", line 575, in call
outputs = self.call(inputs, args, **kwargs)
File "D:\python36\lib\site-packages\tensorflow\python\layers\convolutional.py", line 167, in call
outputs = self._convolution_op(inputs, self.kernel)
File "D:\python36\lib\site-packages\tensorflow\python\ops\nn_ops.py", line 835, in call
return self.conv_op(inp, filter)
File "D:\python36\lib\site-packages\tensorflow\python\ops\nn_ops.py", line 499, in call
return self.call(inp, filter)
File "D:\python36\lib\site-packages\tensorflow\python\ops\nn_ops.py", line 492, in _with_space_to_batch_call
result = self.op(input_converted, filter)
File "D:\python36\lib\site-packages\tensorflow\python\ops\nn_ops.py", line 187, in call
name=self.name)
File "D:\python36\lib\site-packages\tensorflow\python\ops\gen_nn_ops.py", line 630, in conv2d
data_format=data_format, name=name)
File "D:\python36\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "D:\python36\lib\site-packages\tensorflow\python\framework\ops.py", line 2956, in create_op
op_def=op_def)
File "D:\python36\lib\site-packages\tensorflow\python\framework\ops.py", line 1470, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access
My Environment: win10 python 3.6 tensorflow 1.4 gpu gtx965m 2g nvcc -V 8.0 cudnn 6.0 It took me a whole day didn't solve the problem 0.0 ps: Please switch to tf.train.create_global_step INFO:tensorflow:Scale of 0 disables regularizer. INFO:tensorflow:Scale of 0 disables regularizer. INFO:tensorflow:depth of additional conv before box predictor: 0 INFO:tensorflow:Scale of 0 disables regularizer. INFO:tensorflow:Scale of 0 disables regularizer. WARNING:tensorflow:From E:\TensorFlow\TensorFlow Object Detection API Tutorial\models\research\object_detection\meta_architectures\faster_rcnn_meta_arch.py:1670: get_or_create_global_step (from tensorflow.contrib.framework.python.ops.variables) is deprecated and will be removed in a future version. Instructions for updating: Please switch to tf.train.get_or_create_global_step INFO:tensorflow:Summary name /clone_loss is illegal; using clone_loss instead. D:\python36\lib\site-packages\tensorflow\python\ops\gradients_impl.py:96: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory. "Converting sparse IndexedSlices to a dense Tensor of unknown shape. " INFO:tensorflow:Restoring parameters from trainmodels/model.ckpt INFO:tensorflow:Starting Session. INFO:tensorflow:Saving checkpoint to path train/model.ckpt INFO:tensorflow:Starting Queues. INFO:tensorflow:global_step/sec: 0 INFO:tensorflow:Error reported to Coordinator: <class 'tensorflow.python.framework.errors_impl.ResourceExhaustedError'>, OOM when allocating tensor with shape[4,38,56,1088] [[Node: FirstStageFeatureExtractor/InceptionResnetV2/InceptionResnetV2/Repeat_1/block17_3/Conv2d_1x1/Conv2D = Conv2D[T=DT_FLOAT, data_format="NHWC", padding="VALID", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](FirstStageFeatureExtractor/InceptionResnetV2/InceptionResnetV2/Repeat_1/block17_3/Conv2d_1x1/SpaceToBatchND, FirstStageFeatureExtractor/InceptionResnetV2/Repeat_1/block17_3/Conv2d_1x1/weights/read/_945)]] [[Node: BatchMultiClassNonMaxSuppression/map/while/Switch_5/_1739 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_10664_BatchMultiClassNonMaxSuppression/map/while/Switch_5", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
Caused by op 'FirstStageFeatureExtractor/InceptionResnetV2/InceptionResnetV2/Repeat_1/block17_3/Conv2d_1x1/Conv2D', defined at: File "train.py", line 167, in
tf.app.run()
File "D:\python36\lib\site-packages\tensorflow\python\platform\app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "train.py", line 163, in main
worker_job_name, is_chief, FLAGS.train_dir)
File "E:\TensorFlow\TensorFlow Object Detection API Tutorial\models\research\object_detection\trainer.py", line 228, in train
clones = model_deploy.create_clones(deploy_config, model_fn, [input_queue])
File "E:\TensorFlow\TensorFlow Object Detection API Tutorial\models\research\slim\deployment\model_deploy.py", line 193, in create_clones
outputs = model_fn(*args, kwargs)
File "E:\TensorFlow\TensorFlow Object Detection API Tutorial\models\research\object_detection\trainer.py", line 165, in _create_losses
prediction_dict = detection_model.predict(images)
File "E:\TensorFlow\TensorFlow Object Detection API Tutorial\models\research\object_detection\meta_architectures\faster_rcnn_meta_arch.py", line 531, in predict
image_shape) = self._extract_rpn_feature_maps(preprocessed_inputs)
File "E:\TensorFlow\TensorFlow Object Detection API Tutorial\models\research\object_detection\meta_architectures\faster_rcnn_meta_arch.py", line 685, in _extract_rpn_feature_maps
preprocessed_inputs, scope=self.first_stage_feature_extractor_scope)
File "E:\TensorFlow\TensorFlow Object Detection API Tutorial\models\research\object_detection\meta_architectures\faster_rcnn_meta_arch.py", line 134, in extract_proposal_features
return self._extract_proposal_features(preprocessed_inputs, scope)
File "E:\TensorFlow\TensorFlow Object Detection API Tutorial\models\research\object_detection\models\faster_rcnn_inception_resnet_v2_feature_extractor.py", line 112, in _extract_proposal_features
align_feature_maps=True))
File "E:\TensorFlow\TensorFlow Object Detection API Tutorial\models\research\slim\nets\inception_resnet_v2.py", line 244, in inception_resnet_v2_base
activation_fn=activation_fn)
File "D:\python36\lib\site-packages\tensorflow\contrib\layers\python\layers\layers.py", line 2346, in repeat
outputs = layer(outputs, *args, *kwargs)
File "E:\TensorFlow\TensorFlow Object Detection API Tutorial\models\research\slim\nets\inception_resnet_v2.py", line 72, in block17
activation_fn=None, scope='Conv2d_1x1')
File "D:\python36\lib\site-packages\tensorflow\contrib\framework\python\ops\arg_scope.py", line 181, in func_with_args
return func(args, current_args)
File "D:\python36\lib\site-packages\tensorflow\contrib\layers\python\layers\layers.py", line 1033, in convolution
outputs = layer.apply(inputs)
File "D:\python36\lib\site-packages\tensorflow\python\layers\base.py", line 671, in apply
return self.call(inputs, *args, *kwargs)
File "D:\python36\lib\site-packages\tensorflow\python\layers\base.py", line 575, in call
outputs = self.call(inputs, args, **kwargs)
File "D:\python36\lib\site-packages\tensorflow\python\layers\convolutional.py", line 167, in call
outputs = self._convolution_op(inputs, self.kernel)
File "D:\python36\lib\site-packages\tensorflow\python\ops\nn_ops.py", line 835, in call
return self.conv_op(inp, filter)
File "D:\python36\lib\site-packages\tensorflow\python\ops\nn_ops.py", line 499, in call
return self.call(inp, filter)
File "D:\python36\lib\site-packages\tensorflow\python\ops\nn_ops.py", line 492, in _with_space_to_batch_call
result = self.op(input_converted, filter)
File "D:\python36\lib\site-packages\tensorflow\python\ops\nn_ops.py", line 187, in call
name=self.name)
File "D:\python36\lib\site-packages\tensorflow\python\ops\gen_nn_ops.py", line 630, in conv2d
data_format=data_format, name=name)
File "D:\python36\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "D:\python36\lib\site-packages\tensorflow\python\framework\ops.py", line 2956, in create_op
op_def=op_def)
File "D:\python36\lib\site-packages\tensorflow\python\framework\ops.py", line 1470, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access
ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[4,38,56,1088] [[Node: FirstStageFeatureExtractor/InceptionResnetV2/InceptionResnetV2/Repeat_1/block17_3/Conv2d_1x1/Conv2D = Conv2D[T=DT_FLOAT, data_format="NHWC", padding="VALID", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](FirstStageFeatureExtractor/InceptionResnetV2/InceptionResnetV2/Repeat_1/block17_3/Conv2d_1x1/SpaceToBatchND, FirstStageFeatureExtractor/InceptionResnetV2/Repeat_1/block17_3/Conv2d_1x1/weights/read/_945)]] [[Node: BatchMultiClassNonMaxSuppression/map/while/Switch_5/_1739 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_10664_BatchMultiClassNonMaxSuppression/map/while/Switch_5", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
Traceback (most recent call last): File "D:\python36\lib\site-packages\tensorflow\python\client\session.py", line 1323, in _do_call return fn(*args) File "D:\python36\lib\site-packages\tensorflow\python\client\session.py", line 1302, in _run_fn status, run_metadata) File "D:\python36\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 473, in exit c_api.TF_GetCode(self.status.status)) tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[1,75,100,1088] [[Node: FirstStageFeatureExtractor/InceptionResnetV2/InceptionResnetV2/Repeat_1/block17_1/Conv2d_1x1/BatchToSpaceND = BatchToSpaceND[T=DT_FLOAT, Tblock_shape=DT_INT32, Tcrops=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"](FirstStageFeatureExtractor/InceptionResnetV2/InceptionResnetV2/Repeat_1/block17_1/Conv2d_1x1/Conv2D, Conv/SpaceToBatchND/block_shape, FirstStageFeatureExtractor/InceptionResnetV2/InceptionResnetV2/Repeat_1/block17_1/Conv2d_1x1/strided_slice_3)]] [[Node: gradients/SecondStageFeatureExtractor/InceptionResnetV2/Block8/Branch_1/Conv2d_0a_1x1/Conv2D_grad/tuple/control_dependency_1/_4565 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_13255_gradients/SecondStageFeatureExtractor/InceptionResnetV2/Block8/Branch_1/Conv2d_0a_1x1/Conv2D_grad/tuple/control_dependency_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "D:\python36\lib\site-packages\tensorflow\python\training\supervisor.py", line 954, in managed_session yield sess File "D:\python36\lib\site-packages\tensorflow\contrib\slim\python\slim\learning.py", line 763, in train sess, train_op, global_step, train_step_kwargs) File "D:\python36\lib\site-packages\tensorflow\contrib\slim\python\slim\learning.py", line 487, in train_step run_metadata=run_metadata) File "D:\python36\lib\site-packages\tensorflow\python\client\session.py", line 889, in run run_metadata_ptr) File "D:\python36\lib\site-packages\tensorflow\python\client\session.py", line 1120, in _run feed_dict_tensor, options, run_metadata) File "D:\python36\lib\site-packages\tensorflow\python\client\session.py", line 1317, in _do_run options, run_metadata) File "D:\python36\lib\site-packages\tensorflow\python\client\session.py", line 1336, in _do_call raise type(e)(node_def, op, message) tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[1,75,100,1088] [[Node: FirstStageFeatureExtractor/InceptionResnetV2/InceptionResnetV2/Repeat_1/block17_1/Conv2d_1x1/BatchToSpaceND = BatchToSpaceND[T=DT_FLOAT, Tblock_shape=DT_INT32, Tcrops=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"](FirstStageFeatureExtractor/InceptionResnetV2/InceptionResnetV2/Repeat_1/block17_1/Conv2d_1x1/Conv2D, Conv/SpaceToBatchND/block_shape, FirstStageFeatureExtractor/InceptionResnetV2/InceptionResnetV2/Repeat_1/block17_1/Conv2d_1x1/strided_slice_3)]] [[Node: gradients/SecondStageFeatureExtractor/InceptionResnetV2/Block8/Branch_1/Conv2d_0a_1x1/Conv2D_grad/tuple/control_dependency_1/_4565 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_13255_gradients/SecondStageFeatureExtractor/InceptionResnetV2/Block8/Branch_1/Conv2d_0a_1x1/Conv2D_grad/tuple/control_dependency_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Caused by op 'FirstStageFeatureExtractor/InceptionResnetV2/InceptionResnetV2/Repeat_1/block17_1/Conv2d_1x1/BatchToSpaceND', defined at: File "train.py", line 167, in
tf.app.run()
File "D:\python36\lib\site-packages\tensorflow\python\platform\app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "train.py", line 163, in main
worker_job_name, is_chief, FLAGS.train_dir)
File "E:\TensorFlow\TensorFlow Object Detection API Tutorial\models\research\object_detection\trainer.py", line 228, in train
clones = model_deploy.create_clones(deploy_config, model_fn, [input_queue])
File "E:\TensorFlow\TensorFlow Object Detection API Tutorial\models\research\slim\deployment\model_deploy.py", line 193, in create_clones
outputs = model_fn(*args, kwargs)
File "E:\TensorFlow\TensorFlow Object Detection API Tutorial\models\research\object_detection\trainer.py", line 165, in _create_losses
prediction_dict = detection_model.predict(images)
File "E:\TensorFlow\TensorFlow Object Detection API Tutorial\models\research\object_detection\meta_architectures\faster_rcnn_meta_arch.py", line 531, in predict
image_shape) = self._extract_rpn_feature_maps(preprocessed_inputs)
File "E:\TensorFlow\TensorFlow Object Detection API Tutorial\models\research\object_detection\meta_architectures\faster_rcnn_meta_arch.py", line 685, in _extract_rpn_feature_maps
preprocessed_inputs, scope=self.first_stage_feature_extractor_scope)
File "E:\TensorFlow\TensorFlow Object Detection API Tutorial\models\research\object_detection\meta_architectures\faster_rcnn_meta_arch.py", line 134, in extract_proposal_features
return self._extract_proposal_features(preprocessed_inputs, scope)
File "E:\TensorFlow\TensorFlow Object Detection API Tutorial\models\research\object_detection\models\faster_rcnn_inception_resnet_v2_feature_extractor.py", line 112, in _extract_proposal_features
align_feature_maps=True))
File "E:\TensorFlow\TensorFlow Object Detection API Tutorial\models\research\slim\nets\inception_resnet_v2.py", line 244, in inception_resnet_v2_base
activation_fn=activation_fn)
File "D:\python36\lib\site-packages\tensorflow\contrib\layers\python\layers\layers.py", line 2346, in repeat
outputs = layer(outputs, *args, *kwargs)
File "E:\TensorFlow\TensorFlow Object Detection API Tutorial\models\research\slim\nets\inception_resnet_v2.py", line 72, in block17
activation_fn=None, scope='Conv2d_1x1')
File "D:\python36\lib\site-packages\tensorflow\contrib\framework\python\ops\arg_scope.py", line 181, in func_with_args
return func(args, current_args)
File "D:\python36\lib\site-packages\tensorflow\contrib\layers\python\layers\layers.py", line 1033, in convolution
outputs = layer.apply(inputs)
File "D:\python36\lib\site-packages\tensorflow\python\layers\base.py", line 671, in apply
return self.call(inputs, *args, *kwargs)
File "D:\python36\lib\site-packages\tensorflow\python\layers\base.py", line 575, in call
outputs = self.call(inputs, args, **kwargs)
File "D:\python36\lib\site-packages\tensorflow\python\layers\convolutional.py", line 167, in call
outputs = self._convolution_op(inputs, self.kernel)
File "D:\python36\lib\site-packages\tensorflow\python\ops\nn_ops.py", line 835, in call
return self.conv_op(inp, filter)
File "D:\python36\lib\site-packages\tensorflow\python\ops\nn_ops.py", line 499, in call
return self.call(inp, filter)
File "D:\python36\lib\site-packages\tensorflow\python\ops\nn_ops.py", line 495, in _with_space_to_batch_call
input=result, block_shape=dilation_rate, crops=crops)
File "D:\python36\lib\site-packages\tensorflow\python\ops\gen_array_ops.py", line 404, in batch_to_space_nd
name=name)
File "D:\python36\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "D:\python36\lib\site-packages\tensorflow\python\framework\ops.py", line 2956, in create_op
op_def=op_def)
File "D:\python36\lib\site-packages\tensorflow\python\framework\ops.py", line 1470, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access
ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[1,75,100,1088] [[Node: FirstStageFeatureExtractor/InceptionResnetV2/InceptionResnetV2/Repeat_1/block17_1/Conv2d_1x1/BatchToSpaceND = BatchToSpaceND[T=DT_FLOAT, Tblock_shape=DT_INT32, Tcrops=DT_INT32, _device="/job:localhost/replica:0/task:0/device:GPU:0"](FirstStageFeatureExtractor/InceptionResnetV2/InceptionResnetV2/Repeat_1/block17_1/Conv2d_1x1/Conv2D, Conv/SpaceToBatchND/block_shape, FirstStageFeatureExtractor/InceptionResnetV2/InceptionResnetV2/Repeat_1/block17_1/Conv2d_1x1/strided_slice_3)]] [[Node: gradients/SecondStageFeatureExtractor/InceptionResnetV2/Block8/Branch_1/Conv2d_0a_1x1/Conv2D_grad/tuple/control_dependency_1/_4565 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_13255_gradients/SecondStageFeatureExtractor/InceptionResnetV2/Block8/Branch_1/Conv2d_0a_1x1/Conv2D_grad/tuple/control_dependency_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "train.py", line 167, in
tf.app.run()
File "D:\python36\lib\site-packages\tensorflow\python\platform\app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "train.py", line 163, in main
worker_job_name, is_chief, FLAGS.train_dir)
File "E:\TensorFlow\TensorFlow Object Detection API Tutorial\models\research\object_detection\trainer.py", line 337, in train
saver=saver)
File "D:\python36\lib\site-packages\tensorflow\contrib\slim\python\slim\learning.py", line 775, in train
sv.stop(threads, close_summary_writer=True)
File "D:\python36\lib\contextlib.py", line 99, in exit
self.gen.throw(type, value, traceback)
File "D:\python36\lib\site-packages\tensorflow\python\training\supervisor.py", line 964, in managed_session
self.stop(close_summary_writer=close_summary_writer)
File "D:\python36\lib\site-packages\tensorflow\python\training\supervisor.py", line 792, in stop
stop_grace_period_secs=self._stop_grace_secs)
File "D:\python36\lib\site-packages\tensorflow\python\training\coordinator.py", line 389, in join
six.reraise(*self._exc_info_to_raise)
File "D:\python36\lib\site-packages\six.py", line 693, in reraise
raise value
File "D:\python36\lib\site-packages\tensorflow\python\training\coordinator.py", line 296, in stop_on_exception
yield
File "D:\python36\lib\site-packages\tensorflow\python\training\coordinator.py", line 494, in run
self.run_loop()
File "D:\python36\lib\site-packages\tensorflow\python\training\supervisor.py", line 994, in run_loop
self._sv.global_step])
File "D:\python36\lib\site-packages\tensorflow\python\client\session.py", line 889, in run
run_metadata_ptr)
File "D:\python36\lib\site-packages\tensorflow\python\client\session.py", line 1120, in _run
feed_dict_tensor, options, run_metadata)
File "D:\python36\lib\site-packages\tensorflow\python\client\session.py", line 1317, in _do_run
options, run_metadata)
File "D:\python36\lib\site-packages\tensorflow\python\client\session.py", line 1336, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when allocating tensor with shape[4,38,56,1088]
[[Node: FirstStageFeatureExtractor/InceptionResnetV2/InceptionResnetV2/Repeat_1/block17_3/Conv2d_1x1/Conv2D = Conv2D[T=DT_FLOAT, data_format="NHWC", padding="VALID", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](FirstStageFeatureExtractor/InceptionResnetV2/InceptionResnetV2/Repeat_1/block17_3/Conv2d_1x1/SpaceToBatchND, FirstStageFeatureExtractor/InceptionResnetV2/Repeat_1/block17_3/Conv2d_1x1/weights/read/_945)]]
[[Node: BatchMultiClassNonMaxSuppression/map/while/Switch_5/_1739 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_10664_BatchMultiClassNonMaxSuppression/map/while/Switch_5", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]
Caused by op 'FirstStageFeatureExtractor/InceptionResnetV2/InceptionResnetV2/Repeat_1/block17_3/Conv2d_1x1/Conv2D', defined at: File "train.py", line 167, in
tf.app.run()
File "D:\python36\lib\site-packages\tensorflow\python\platform\app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "train.py", line 163, in main
worker_job_name, is_chief, FLAGS.train_dir)
File "E:\TensorFlow\TensorFlow Object Detection API Tutorial\models\research\object_detection\trainer.py", line 228, in train
clones = model_deploy.create_clones(deploy_config, model_fn, [input_queue])
File "E:\TensorFlow\TensorFlow Object Detection API Tutorial\models\research\slim\deployment\model_deploy.py", line 193, in create_clones
outputs = model_fn(*args, kwargs)
File "E:\TensorFlow\TensorFlow Object Detection API Tutorial\models\research\object_detection\trainer.py", line 165, in _create_losses
prediction_dict = detection_model.predict(images)
File "E:\TensorFlow\TensorFlow Object Detection API Tutorial\models\research\object_detection\meta_architectures\faster_rcnn_meta_arch.py", line 531, in predict
image_shape) = self._extract_rpn_feature_maps(preprocessed_inputs)
File "E:\TensorFlow\TensorFlow Object Detection API Tutorial\models\research\object_detection\meta_architectures\faster_rcnn_meta_arch.py", line 685, in _extract_rpn_feature_maps
preprocessed_inputs, scope=self.first_stage_feature_extractor_scope)
File "E:\TensorFlow\TensorFlow Object Detection API Tutorial\models\research\object_detection\meta_architectures\faster_rcnn_meta_arch.py", line 134, in extract_proposal_features
return self._extract_proposal_features(preprocessed_inputs, scope)
File "E:\TensorFlow\TensorFlow Object Detection API Tutorial\models\research\object_detection\models\faster_rcnn_inception_resnet_v2_feature_extractor.py", line 112, in _extract_proposal_features
align_feature_maps=True))
File "E:\TensorFlow\TensorFlow Object Detection API Tutorial\models\research\slim\nets\inception_resnet_v2.py", line 244, in inception_resnet_v2_base
activation_fn=activation_fn)
File "D:\python36\lib\site-packages\tensorflow\contrib\layers\python\layers\layers.py", line 2346, in repeat
outputs = layer(outputs, *args, *kwargs)
File "E:\TensorFlow\TensorFlow Object Detection API Tutorial\models\research\slim\nets\inception_resnet_v2.py", line 72, in block17
activation_fn=None, scope='Conv2d_1x1')
File "D:\python36\lib\site-packages\tensorflow\contrib\framework\python\ops\arg_scope.py", line 181, in func_with_args
return func(args, current_args)
File "D:\python36\lib\site-packages\tensorflow\contrib\layers\python\layers\layers.py", line 1033, in convolution
outputs = layer.apply(inputs)
File "D:\python36\lib\site-packages\tensorflow\python\layers\base.py", line 671, in apply
return self.call(inputs, *args, *kwargs)
File "D:\python36\lib\site-packages\tensorflow\python\layers\base.py", line 575, in call
outputs = self.call(inputs, args, **kwargs)
File "D:\python36\lib\site-packages\tensorflow\python\layers\convolutional.py", line 167, in call
outputs = self._convolution_op(inputs, self.kernel)
File "D:\python36\lib\site-packages\tensorflow\python\ops\nn_ops.py", line 835, in call
return self.conv_op(inp, filter)
File "D:\python36\lib\site-packages\tensorflow\python\ops\nn_ops.py", line 499, in call
return self.call(inp, filter)
File "D:\python36\lib\site-packages\tensorflow\python\ops\nn_ops.py", line 492, in _with_space_to_batch_call
result = self.op(input_converted, filter)
File "D:\python36\lib\site-packages\tensorflow\python\ops\nn_ops.py", line 187, in call
name=self.name)
File "D:\python36\lib\site-packages\tensorflow\python\ops\gen_nn_ops.py", line 630, in conv2d
data_format=data_format, name=name)
File "D:\python36\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "D:\python36\lib\site-packages\tensorflow\python\framework\ops.py", line 2956, in create_op
op_def=op_def)
File "D:\python36\lib\site-packages\tensorflow\python\framework\ops.py", line 1470, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access
ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[4,38,56,1088] [[Node: FirstStageFeatureExtractor/InceptionResnetV2/InceptionResnetV2/Repeat_1/block17_3/Conv2d_1x1/Conv2D = Conv2D[T=DT_FLOAT, data_format="NHWC", padding="VALID", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](FirstStageFeatureExtractor/InceptionResnetV2/InceptionResnetV2/Repeat_1/block17_3/Conv2d_1x1/SpaceToBatchND, FirstStageFeatureExtractor/InceptionResnetV2/Repeat_1/block17_3/Conv2d_1x1/weights/read/_945)]] [[Node: BatchMultiClassNonMaxSuppression/map/while/Switch_5/_1739 = _Recvclient_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_10664_BatchMultiClassNonMaxSuppression/map/while/Switch_5", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]]