when i training my data,i get a error(ValueError: cannot convert float NaN to integer).i dont konw why?

[gavinzhang1995]

I1114 21:43:12.214639 28872 sgd_solver.cpp:106] Iteration 0, lr = 0.0002 /home/gavin/icpr/FPN-caffe/tools/../lib/rpn/proposal_target_layer.py:225: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future fg_inds = npr.choice(fg_inds, size=fg_rois_per_this_image, replace=False) /home/gavin/icpr/FPN-caffe/tools/../lib/rpn/proposal_target_layer.py:236: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future bg_inds = npr.choice(bg_inds, size=bg_rois_per_this_image, replace=False) /home/gavin/icpr/FPN-caffe/tools/../lib/rpn/proposal_target_layer.py:243: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future labels[fg_rois_per_this_image:] = 0 /home/gavin/icpr/FPN-caffe/tools/../lib/rpn/anchor_target_layer.py:104: RuntimeWarning: invalid value encountered in sqrt k = np.floor(k0 + np.log2(np.sqrt(areas)*1.0 / 64)) /home/gavin/icpr/FPN-caffe/tools/../lib/rpn/anchor_target_layer.py:106: RuntimeWarning: invalid value encountered in less inds = np.where(k < min_k)[0] /home/gavin/icpr/FPN-caffe/tools/../lib/rpn/anchor_target_layer.py:108: RuntimeWarning: invalid value encountered in greater inds = np.where(k > max_k)[0] Traceback (most recent call last): File "./tools/train_net.py", line 112, in max_iters=args.max_iters) File "/home/gavin/icpr/FPN-caffe/tools/../lib/fast_rcnn/train.py", line 160, in train_net model_paths = sw.train_model(max_iters) File "/home/gavin/icpr/FPN-caffe/tools/../lib/fast_rcnn/train.py", line 101, in train_model self.solver.step(1) File "/home/gavin/icpr/FPN-caffe/tools/../lib/rpn/anchor_target_layer.py", line 116, in forward if int(item)==p: ValueError: cannot convert float NaN to integer Command exited with non-zero status 1

[busyboxs]

@gavinzhang1995
as to VisibleDeprecationWarning, maybe you should change code as follow: fg_inds = npr.choice(fg_inds, size=int(fg_rois_per_this_image), replace=False) bg_inds = npr.choice(bg_inds, size=int(bg_rois_per_this_image), replace=False) labels[int(fg_rois_per_this_image):] = 0

[cs-heibao]

@gavinzhang1995 Did you solve the error?

[cs-heibao]

@busyboxs @gavinzhang1995 I found that the error caused by this line of code “ k = np.floor(k0 + np.log2(np.sqrt(areas)1.0 / 64))"，and the mainly problem is that the value of areas sometimes maybe negtive, it's very strange！ The error occurred when Iteration 200, and I print the areas： I1117 21:20:22.956712 20135 solver.cpp:229] Iteration 200, loss = 0.981587 I1117 21:20:22.956739 20135 solver.cpp:245] Train net output #0: loss_bbox = 0.0910424 ( 1 = 0.0910424 loss) I1117 21:20:22.956758 20135 solver.cpp:245] Train net output #1: loss_cls = 0.268306 ( 1 = 0.268306 loss) I1117 21:20:22.956763 20135 solver.cpp:245] Train net output #2: rpn_cls_loss/p2 = 0.0182713 ( 1 = 0.0182713 loss) I1117 21:20:22.956768 20135 solver.cpp:245] Train net output #3: rpn_cls_loss/p3 = 0.107832 ( 1 = 0.107832 loss) I1117 21:20:22.956784 20135 solver.cpp:245] Train net output #4: rpn_cls_loss/p4 = 0.0701596 ( 1 = 0.0701596 loss) I1117 21:20:22.956787 20135 solver.cpp:245] Train net output #5: rpn_cls_loss/p5 = 0.128425 ( 1 = 0.128425 loss) I1117 21:20:22.956806 20135 solver.cpp:245] Train net output #6: rpn_loss_bbox/p2 = 0 ( 1 = 0 loss) I1117 21:20:22.956810 20135 solver.cpp:245] Train net output #7: rpn_loss_bbox/p3 = 0.00275289 ( 1 = 0.00275289 loss) I1117 21:20:22.956815 20135 solver.cpp:245] Train net output #8: rpn_loss_bbox/p4 = 0.00327946 ( 1 = 0.00327946 loss) I1117 21:20:22.956818 20135 solver.cpp:245] Train net output #9: rpn_loss_bbox/p5 = 0 (* 1 = 0 loss) I1117 21:20:22.956823 20135 sgd_solver.cpp:106] Iteration 200, lr = 0.0002 [ 98314.6796875] [ 4973.14453125 1771.54541016 1843.26171875 1879.8828125 4442.74902344] [ 4973.14453125 1771.54541016 1843.26171875 1879.8828125 4442.74902344] [-6696563.] and there is a negative number, I don't know why

[busyboxs]

@junjieAI I trained network with pascal voc dataset, and there is no problem. Which dataset did you use?

[cs-heibao]

@busyboxs You used the public pascal voc data or your own data？ I used my own data

[RoseLii]

Did you solve the error? @gavinzhang1995 I also had this problem.

[busyboxs]

@RoseLii what about change TRAIN.RPN_MIN_SIZE = 16 in "lib/fast_rcnn/config.py" or set it in "FP_Net_end2end.yml", you can have a try.

[RoseLii]

thank you.i will try the way. @busyboxs

[RoseLii]

@busyboxs excuse me.我在训练VOC2007时，整个训练过程是没有问题的，但是我在训练自己的数据集时出现了这个问题。由于我的数据集中有些图片是没有标签的，所以我将那部分的标签设为0 0 0 0，然后生成xml文件，然后就报错了，不知道怎么改 @junjieAI 你的问题解决了吗？帮帮我

[gavinzhang1995]

so sorry,i dont solve this error.i also use my own data. @junjieAI @busyboxs @RoseLii @unsky

[unsky]

i have update codes

[sujinpeng]

Did you solve the error? @gavinzhang1995 I meet the problem,thank you

[sujinpeng]

could you tell me what is 64?why not 224?