search

gklz1982 / caffe-yolov2

Other
164 stars 128 forks source link

error appear at the begining. #5

Open OPPOA113 opened 7 years ago

OPPOA113 commented 7 years ago

I0804 17:49:41.090296 13616 solver.cpp:245] Train net output #0: region1 = 130.248 ( 1 = 130.248 loss) I0804 17:49:41.090306 13616 sgd_solver.cpp:115] Iteration 0, lr = 0.0001 I0804 17:50:22.566540 13616 region_loss_layer.cpp:469] avg_noobj: 0.440657 avg_obj: 0.441241 avg_iou: 0.623299 avg_cat: 0.250071 recall: 1 class_count: 5 I0804 17:50:53.259757 13616 solver.cpp:229] Iteration 20, loss = 104.177 I0804 17:50:53.259923 13616 solver.cpp:245] Train net output #0: region1 = 96.0165 ( 1 = 96.0165 loss) I0804 17:50:53.259968 13616 sgd_solver.cpp:115] Iteration 20, lr = 0.0001 I0804 17:51:07.790935 13616 region_loss_layer.cpp:469] avg_noobj: 0.348465 avg_obj: 0.348111 avg_iou: 0.431767 avg_cat: 0.250216 recall: 0.391304 class_count: 23 I0804 17:51:53.122269 13616 region_loss_layer.cpp:469] avg_noobj: 0.270268 avg_obj: 0.270857 avg_iou: 0.55787 avg_cat: 0.24985 recall: 0.857143 class_count: 7 I0804 17:52:06.210116 13616 solver.cpp:229] Iteration 40, loss = 49.2193 I0804 17:52:06.210199 13616 solver.cpp:245] Train net output #0: region1 = 41.1888 ( 1 = 41.1888 loss) I0804 17:52:06.210211 13616 sgd_solver.cpp:115] Iteration 40, lr = 0.0001 I0804 17:52:39.102437 13616 region_loss_layer.cpp:469] avg_noobj: 0.223499 avg_obj: 0.225008 avg_iou: 0.568238 avg_cat: 0.250957 recall: 0.666667 class_count: 6 F0804 17:53:05.186326 13616 region_loss_layer.cpp:234] Check failed: ## swap_data[index] >= 0 (-nan vs. 0) Check failure stack trace: ***** @ 0x7f2e06772daa (unknown)

why the value below 0 ?

zheLim commented 7 years ago

Yes i have the same problem. Swap_data becomes an -nan. In my case, one of my data label goes wrong. ymin = ymax, which means w=0. divide to 0 can cause nan.

popper0912 commented 6 years ago

how to add negative data?

skyuuka commented 5 years ago

Yes i have the same problem. Swap_data becomes an -nan. In my case, one of my data label goes wrong. ymin = ymax, which means w=0. divide to 0 can cause nan.

I also met the same issue, and in my case it also seems to be data problem. I found some of my data has swapped min and max or the coordinates. After fixing that, it works again.

xincxiong commented 5 years ago

I0804 17:49:41.090296 13616 solver.cpp:245] Train net output #0: region1 = 130.248 ( 1 = 130.248 loss) I0804 17:49:41.090306 13616 sgd_solver.cpp:115] Iteration 0, lr = 0.0001 I0804 17:50:22.566540 13616 region_loss_layer.cpp:469] avg_noobj: 0.440657 avg_obj: 0.441241 avg_iou: 0.623299 avg_cat: 0.250071 recall: 1 class_count: 5 I0804 17:50:53.259757 13616 solver.cpp:229] Iteration 20, loss = 104.177 I0804 17:50:53.259923 13616 solver.cpp:245] Train net output #0: region1 = 96.0165 ( 1 = 96.0165 loss) I0804 17:50:53.259968 13616 sgd_solver.cpp:115] Iteration 20, lr = 0.0001 I0804 17:51:07.790935 13616 region_loss_layer.cpp:469] avg_noobj: 0.348465 avg_obj: 0.348111 avg_iou: 0.431767 avg_cat: 0.250216 recall: 0.391304 class_count: 23 I0804 17:51:53.122269 13616 region_loss_layer.cpp:469] avg_noobj: 0.270268 avg_obj: 0.270857 avg_iou: 0.55787 avg_cat: 0.24985 recall: 0.857143 class_count: 7 I0804 17:52:06.210116 13616 solver.cpp:229] Iteration 40, loss = 49.2193 I0804 17:52:06.210199 13616 solver.cpp:245] Train net output #0: region1 = 41.1888 ( 1 = 41.1888 loss) I0804 17:52:06.210211 13616 sgd_solver.cpp:115] Iteration 40, lr = 0.0001 I0804 17:52:39.102437 13616 region_loss_layer.cpp:469] avg_noobj: 0.223499 avg_obj: 0.225008 avg_iou: 0.568238 avg_cat: 0.250957 recall: 0.666667 class_count: 6 F0804 17:53:05.186326 13616 region_loss_layer.cpp:234] Check failed: ## swap_data[index] >= 0 (-nan vs. 0) Check failure stack trace: ***** @ 0x7f2e06772daa (unknown)

why the value below 0 ?

I0327 05:32:30.823518 16831 solver.cpp:218] Iteration 200 (3.38936 iter/s, 59.0082s/200 iters), loss = inf I0327 05:32:30.823693 16831 solver.cpp:237] Train net output #0: det_loss = inf (* 1 = inf loss) I0327 05:32:30.823709 16831 sgd_solver.cpp:114] Iteration 200, lr = 0.0001 F0327 05:32:30.917165 16831 region_loss_layer.cpp:230] Check failed: swap_data[index] >= 0 (-nan vs. 0) Check failure stack trace: @ 0x7fb37d19d5cd google::LogMessage::Fail() @ 0x7fb37d19f433 google::LogMessage::SendToLog() @ 0x7fb37d19d15b google::LogMessage::Flush() @ 0x7fb37d19fe1e google::LogMessageFatal::~LogMessageFatal() @ 0x7fb37d818144 caffe::RegionLossLayer<>::Forward_cpu() @ 0x7fb37d7e5857 caffe::Net<>::ForwardFromTo() @ 0x7fb37d7e5be7 caffe::Net<>::Forward() @ 0x7fb37d97caa8 caffe::Solver<>::Step() @ 0x7fb37d97d64a caffe::Solver<>::Solve() @ 0x40f1e5 train() @ 0x40b4bd main @ 0x7fb37b75b830 __libc_start_main @ 0x40c049 _start @ (nil) (unknown) Aborted (core dumped)

Does anyone know how to solve it?