F:\1xjie\Fasterrcnn\lib\utils\bbox_transform.py:52: RuntimeWarning: overflow encountered in exp
pred_w = np.exp(dw) widths[:, np.newaxis]
F:\1xjie\Fasterrcnn\lib\utils\bbox_transform.py:52: RuntimeWarning: overflow encountered in multiply
pred_w = np.exp(dw) widths[:, np.newaxis]
F:\1xjie\Fasterrcnn\lib\utils\bbox_transform.py:53: RuntimeWarning: overflow encountered in exp
pred_h = np.exp(dh) heights[:, np.newaxis]
F:\1xjie\Fasterrcnn\lib\utils\bbox_transform.py:53: RuntimeWarning: overflow encountered in multiply
pred_h = np.exp(dh) heights[:, np.newaxis]
pdb.set_trace
Traceback (most recent call last):
File "C:\Users\Administrator\Anaconda3\envs\fasterrcnn35\lib\site-packages\tensorflow\python\client\session.py", line 1350, in _do_call
return fn(*args)
File "C:\Users\Administrator\Anaconda3\envs\fasterrcnn35\lib\site-packages\tensorflow\python\client\session.py", line 1329, in _run_fn
status, run_metadata)
File "C:\Users\Administrator\Anaconda3\envs\fasterrcnn35\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 473, in exit
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: Incompatible shapes: [64,8] vs. [0,8]
[[Node: gradients/loss_default/mul_9_grad/Mul_1 = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](gradients/loss_default/Sum_1_grad/Tile, vgg_16/rpn_rois/PyFunc/_325)]]
[[Node: gradients/vgg_16/conv4/conv4_3/Conv2D_grad/tuple/control_dependency_1/_383 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1067_gradients/vgg_16/conv4/conv4_3/Conv2D_grad/tuple/control_dependency_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "F:/1xjie/Fasterrcnn/trainWXJ.py", line 258, in
train.train()
File "F:/1xjie/Fasterrcnn/trainWXJ.py", line 164, in train
rpn_loss_cls, rpn_loss_box, loss_cls, loss_box, total_loss = self.net.train_step(sess, blobs, train_op)
File "F:\1xjie\Fasterrcnn\lib\nets\network.py", line 373, in train_step
feed_dict=feed_dict)
File "C:\Users\Administrator\Anaconda3\envs\fasterrcnn35\lib\site-packages\tensorflow\python\client\session.py", line 895, in run
run_metadata_ptr)
File "C:\Users\Administrator\Anaconda3\envs\fasterrcnn35\lib\site-packages\tensorflow\python\client\session.py", line 1128, in _run
feed_dict_tensor, options, run_metadata)
File "C:\Users\Administrator\Anaconda3\envs\fasterrcnn35\lib\site-packages\tensorflow\python\client\session.py", line 1344, in _do_run
options, run_metadata)
File "C:\Users\Administrator\Anaconda3\envs\fasterrcnn35\lib\site-packages\tensorflow\python\client\session.py", line 1363, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Incompatible shapes: [64,8] vs. [0,8]
[[Node: gradients/loss_default/mul_9_grad/Mul_1 = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](gradients/loss_default/Sum_1_grad/Tile, vgg_16/rpn_rois/PyFunc/_325)]]
[[Node: gradients/vgg_16/conv4/conv4_3/Conv2D_grad/tuple/control_dependency_1/_383 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1067_gradients/vgg_16/conv4/conv4_3/Conv2D_grad/tuple/control_dependency_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Caused by op 'gradients/loss_default/mul_9_grad/Mul_1', defined at:
File "F:/1xjie/Fasterrcnn/trainWXJ.py", line 258, in
train.train()
File "F:/1xjie/Fasterrcnn/trainWXJ.py", line 103, in train
gvs = optimizer.compute_gradients(loss)
File "C:\Users\Administrator\Anaconda3\envs\fasterrcnn35\lib\site-packages\tensorflow\python\training\optimizer.py", line 456, in compute_gradients
colocate_gradients_with_ops=colocate_gradients_with_ops)
File "C:\Users\Administrator\Anaconda3\envs\fasterrcnn35\lib\site-packages\tensorflow\python\ops\gradients_impl.py", line 609, in gradients
grad_scope, op, func_call, lambda: grad_fn(op, out_grads))
File "C:\Users\Administrator\Anaconda3\envs\fasterrcnn35\lib\site-packages\tensorflow\python\ops\gradients_impl.py", line 375, in _MaybeCompile
return grad_fn() # Exit early
File "C:\Users\Administrator\Anaconda3\envs\fasterrcnn35\lib\site-packages\tensorflow\python\ops\gradients_impl.py", line 609, in
grad_scope, op, func_call, lambda: grad_fn(op, out_grads))
File "C:\Users\Administrator\Anaconda3\envs\fasterrcnn35\lib\site-packages\tensorflow\python\ops\math_grad.py", line 779, in _MulGrad
return gen_math_ops._mul(grad, y), gen_math_ops._mul(grad, x)
File "C:\Users\Administrator\Anaconda3\envs\fasterrcnn35\lib\site-packages\tensorflow\python\ops\gen_math_ops.py", line 3100, in _mul
"Mul", x=x, y=y, name=name)
File "C:\Users\Administrator\Anaconda3\envs\fasterrcnn35\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "C:\Users\Administrator\Anaconda3\envs\fasterrcnn35\lib\site-packages\tensorflow\python\framework\ops.py", line 3160, in create_op
op_def=op_def)
File "C:\Users\Administrator\Anaconda3\envs\fasterrcnn35\lib\site-packages\tensorflow\python\framework\ops.py", line 1625, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access
...which was originally created as op 'loss_default/mul_9', defined at:
File "F:/1xjie/Fasterrcnn/trainWXJ.py", line 258, in
train.train()
File "F:/1xjie/Fasterrcnn/trainWXJ.py", line 97, in train
layers = self.net.create_architecture(sess, "TRAIN", self.imdb.num_classes, tag='default')
File "F:\1xjie\Fasterrcnn\lib\nets\network.py", line 311, in create_architecture
self._add_losses()
File "F:\1xjie\Fasterrcnn\lib\nets\network.py", line 249, in _add_losses
loss_box = self._smooth_l1_loss(bbox_pred, bbox_targets, bbox_inside_weights, bbox_outside_weights)
File "F:\1xjie\Fasterrcnn\lib\nets\network.py", line 208, in _smooth_l1_loss
out_loss_box = bbox_outside_weights * in_loss_box
File "C:\Users\Administrator\Anaconda3\envs\fasterrcnn35\lib\site-packages\tensorflow\python\ops\math_ops.py", line 907, in binary_op_wrapper
return func(x, y, name=name)
File "C:\Users\Administrator\Anaconda3\envs\fasterrcnn35\lib\site-packages\tensorflow\python\ops\math_ops.py", line 1131, in _mul_dispatch
return gen_math_ops._mul(x, y, name=name)
File "C:\Users\Administrator\Anaconda3\envs\fasterrcnn35\lib\site-packages\tensorflow\python\ops\gen_math_ops.py", line 3100, in _mul
"Mul", x=x, y=y, name=name)
InvalidArgumentError (see above for traceback): Incompatible shapes: [64,8] vs. [0,8]
[[Node: gradients/loss_default/mul_9_grad/Mul_1 = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](gradients/loss_default/Sum_1_grad/Tile, vgg_16/rpn_rois/PyFunc/_325)]]
[[Node: gradients/vgg_16/conv4/conv4_3/Conv2D_grad/tuple/control_dependency_1/_383 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1067_gradients/vgg_16/conv4/conv4_3/Conv2D_grad/tuple/control_dependency_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
When I train, an error happend. What can I do? Thank you !
F:\1xjie\Fasterrcnn\lib\utils\bbox_transform.py:52: RuntimeWarning: overflow encountered in exp pred_w = np.exp(dw) widths[:, np.newaxis] F:\1xjie\Fasterrcnn\lib\utils\bbox_transform.py:52: RuntimeWarning: overflow encountered in multiply pred_w = np.exp(dw) widths[:, np.newaxis] F:\1xjie\Fasterrcnn\lib\utils\bbox_transform.py:53: RuntimeWarning: overflow encountered in exp pred_h = np.exp(dh) heights[:, np.newaxis] F:\1xjie\Fasterrcnn\lib\utils\bbox_transform.py:53: RuntimeWarning: overflow encountered in multiply pred_h = np.exp(dh) heights[:, np.newaxis] pdb.set_trace Traceback (most recent call last): File "C:\Users\Administrator\Anaconda3\envs\fasterrcnn35\lib\site-packages\tensorflow\python\client\session.py", line 1350, in _do_call return fn(*args) File "C:\Users\Administrator\Anaconda3\envs\fasterrcnn35\lib\site-packages\tensorflow\python\client\session.py", line 1329, in _run_fn status, run_metadata) File "C:\Users\Administrator\Anaconda3\envs\fasterrcnn35\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 473, in exit c_api.TF_GetCode(self.status.status)) tensorflow.python.framework.errors_impl.InvalidArgumentError: Incompatible shapes: [64,8] vs. [0,8] [[Node: gradients/loss_default/mul_9_grad/Mul_1 = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](gradients/loss_default/Sum_1_grad/Tile, vgg_16/rpn_rois/PyFunc/_325)]] [[Node: gradients/vgg_16/conv4/conv4_3/Conv2D_grad/tuple/control_dependency_1/_383 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1067_gradients/vgg_16/conv4/conv4_3/Conv2D_grad/tuple/control_dependency_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last): File "F:/1xjie/Fasterrcnn/trainWXJ.py", line 258, in
train.train()
File "F:/1xjie/Fasterrcnn/trainWXJ.py", line 164, in train
rpn_loss_cls, rpn_loss_box, loss_cls, loss_box, total_loss = self.net.train_step(sess, blobs, train_op)
File "F:\1xjie\Fasterrcnn\lib\nets\network.py", line 373, in train_step
feed_dict=feed_dict)
File "C:\Users\Administrator\Anaconda3\envs\fasterrcnn35\lib\site-packages\tensorflow\python\client\session.py", line 895, in run
run_metadata_ptr)
File "C:\Users\Administrator\Anaconda3\envs\fasterrcnn35\lib\site-packages\tensorflow\python\client\session.py", line 1128, in _run
feed_dict_tensor, options, run_metadata)
File "C:\Users\Administrator\Anaconda3\envs\fasterrcnn35\lib\site-packages\tensorflow\python\client\session.py", line 1344, in _do_run
options, run_metadata)
File "C:\Users\Administrator\Anaconda3\envs\fasterrcnn35\lib\site-packages\tensorflow\python\client\session.py", line 1363, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Incompatible shapes: [64,8] vs. [0,8]
[[Node: gradients/loss_default/mul_9_grad/Mul_1 = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](gradients/loss_default/Sum_1_grad/Tile, vgg_16/rpn_rois/PyFunc/_325)]]
[[Node: gradients/vgg_16/conv4/conv4_3/Conv2D_grad/tuple/control_dependency_1/_383 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1067_gradients/vgg_16/conv4/conv4_3/Conv2D_grad/tuple/control_dependency_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Caused by op 'gradients/loss_default/mul_9_grad/Mul_1', defined at: File "F:/1xjie/Fasterrcnn/trainWXJ.py", line 258, in
train.train()
File "F:/1xjie/Fasterrcnn/trainWXJ.py", line 103, in train
gvs = optimizer.compute_gradients(loss)
File "C:\Users\Administrator\Anaconda3\envs\fasterrcnn35\lib\site-packages\tensorflow\python\training\optimizer.py", line 456, in compute_gradients
colocate_gradients_with_ops=colocate_gradients_with_ops)
File "C:\Users\Administrator\Anaconda3\envs\fasterrcnn35\lib\site-packages\tensorflow\python\ops\gradients_impl.py", line 609, in gradients
grad_scope, op, func_call, lambda: grad_fn(op, out_grads))
File "C:\Users\Administrator\Anaconda3\envs\fasterrcnn35\lib\site-packages\tensorflow\python\ops\gradients_impl.py", line 375, in _MaybeCompile
return grad_fn() # Exit early
File "C:\Users\Administrator\Anaconda3\envs\fasterrcnn35\lib\site-packages\tensorflow\python\ops\gradients_impl.py", line 609, in
grad_scope, op, func_call, lambda: grad_fn(op, out_grads))
File "C:\Users\Administrator\Anaconda3\envs\fasterrcnn35\lib\site-packages\tensorflow\python\ops\math_grad.py", line 779, in _MulGrad
return gen_math_ops._mul(grad, y), gen_math_ops._mul(grad, x)
File "C:\Users\Administrator\Anaconda3\envs\fasterrcnn35\lib\site-packages\tensorflow\python\ops\gen_math_ops.py", line 3100, in _mul
"Mul", x=x, y=y, name=name)
File "C:\Users\Administrator\Anaconda3\envs\fasterrcnn35\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "C:\Users\Administrator\Anaconda3\envs\fasterrcnn35\lib\site-packages\tensorflow\python\framework\ops.py", line 3160, in create_op
op_def=op_def)
File "C:\Users\Administrator\Anaconda3\envs\fasterrcnn35\lib\site-packages\tensorflow\python\framework\ops.py", line 1625, in init
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access
...which was originally created as op 'loss_default/mul_9', defined at: File "F:/1xjie/Fasterrcnn/trainWXJ.py", line 258, in
train.train()
File "F:/1xjie/Fasterrcnn/trainWXJ.py", line 97, in train
layers = self.net.create_architecture(sess, "TRAIN", self.imdb.num_classes, tag='default')
File "F:\1xjie\Fasterrcnn\lib\nets\network.py", line 311, in create_architecture
self._add_losses()
File "F:\1xjie\Fasterrcnn\lib\nets\network.py", line 249, in _add_losses
loss_box = self._smooth_l1_loss(bbox_pred, bbox_targets, bbox_inside_weights, bbox_outside_weights)
File "F:\1xjie\Fasterrcnn\lib\nets\network.py", line 208, in _smooth_l1_loss
out_loss_box = bbox_outside_weights * in_loss_box
File "C:\Users\Administrator\Anaconda3\envs\fasterrcnn35\lib\site-packages\tensorflow\python\ops\math_ops.py", line 907, in binary_op_wrapper
return func(x, y, name=name)
File "C:\Users\Administrator\Anaconda3\envs\fasterrcnn35\lib\site-packages\tensorflow\python\ops\math_ops.py", line 1131, in _mul_dispatch
return gen_math_ops._mul(x, y, name=name)
File "C:\Users\Administrator\Anaconda3\envs\fasterrcnn35\lib\site-packages\tensorflow\python\ops\gen_math_ops.py", line 3100, in _mul
"Mul", x=x, y=y, name=name)
InvalidArgumentError (see above for traceback): Incompatible shapes: [64,8] vs. [0,8] [[Node: gradients/loss_default/mul_9_grad/Mul_1 = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](gradients/loss_default/Sum_1_grad/Tile, vgg_16/rpn_rois/PyFunc/_325)]] [[Node: gradients/vgg_16/conv4/conv4_3/Conv2D_grad/tuple/control_dependency_1/_383 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_1067_gradients/vgg_16/conv4/conv4_3/Conv2D_grad/tuple/control_dependency_1", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
When I train, an error happend. What can I do? Thank you !