search

endernewton / tf-faster-rcnn

Tensorflow Faster RCNN for Object Detection
https://arxiv.org/pdf/1702.02138.pdf
MIT License
3.65k stars 1.57k forks source link

Nan in summary histogram when train Faster R-CNN on my own dataset? #314

Open weisq2691 opened 6 years ago

weisq2691 commented 6 years ago

Caused by op u'SCORE/resnet_v1_101_3/anchor/anchor_target/rpn_bbox_targets/scores', defined at: File "./tools/trainval_net.py", line 139, in max_iters=args.max_iters) File "/home/w/tf-faster-rcnn/tools/../lib/model/train_val.py", line 377, in train_net sw.train_model(sess, max_iters) File "/home/w/tf-faster-rcnn/tools/../lib/model/train_val.py", line 248, in train_model lr, train_op = self.construct_graph(sess) File "/home/w/tf-faster-rcnn/tools/../lib/model/train_val.py", line 123, in construct_graph anchor_ratios=cfg.ANCHOR_RATIOS) File "/home/w/tf-faster-rcnn/tools/../lib/nets/network.py", line 423, in create_architecture self._add_score_summary(key, var) File "/home/w/tf-faster-rcnn/tools/../lib/nets/network.py", line 63, in _add_score_summary tf.summary.histogram('SCORE/' + tensor.op.name + '/' + key + '/scores', tensor) File "/home/w/.local/lib/python2.7/site-packages/tensorflow/python/summary/summary.py", line 221, in histogram tag=scope.rstrip('/'), values=values, name=scope) File "/home/w/.local/lib/python2.7/site-packages/tensorflow/python/ops/gen_logging_ops.py", line 131, in _histogram_summary name=name) File "/home/w/.local/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 767, in apply_op op_def=op_def) File "/home/w/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2506, in create_op original_op=self._default_original_op, op_def=op_def) File "/home/w/.local/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1269, in init self._traceback = _extract_stack()

InvalidArgumentError (see above for traceback): Nan in summary histogram for: SCORE/resnet_v1_101_3/anchor/anchor_target/rpn_bbox_targets/scores [[Node: SCORE/resnet_v1_101_3/anchor/anchor_target/rpn_bbox_targets/scores = HistogramSummary[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"](SCORE/resnet_v1_101_3/anchor/anchor_target/rpn_bbox_targets/scores/tag, resnet_v1_101_3/anchor/anchor_target:1)]] [[Node: resnet_v1_101_3/rpn_rois/proposal_target/_1403 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/gpu:0", send_device="/job:localhost/replica:0/task:0/cpu:0", send_device_incarnation=1, tensor_name="edge_7092_resnet_v1_101_3/rpn_rois/proposal_target", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/gpu:0"]()]]

Command exited with non-zero status 1 19.26user 1.47system 0:18.07elapsed 114%CPU (0avgtext+0avgdata 1779808maxresident)k 0inputs+15568outputs (0major+598802minor)pagefaults 0swaps

andytung2019 commented 6 years ago

weisq2691 , I have the same error! Have you found the solution?

chanyixialex commented 6 years ago

weisq2691 , I have the same error! Have you found the solution?

jplnasa5 commented 5 years ago

I have the same error! Have you found the solution?

devendraswamy commented 4 years ago

I have the same error! Have you found the solution ? i have to try run my own dataset , after some iterations i am getting same problem , please help me to solve that problem