the loss declined so slowly!

[justttry]

i train the model on RCTW-17 and some doc pictures, but the loss declined so slowly. I have spent 12 hours, and the the model was trained only 3000 iterations. The loss is from 4.71 to 3.47 in 12 hours. TOO SLOW! Is there anything wrong?

below is the print.

I0320 18:24:14.414151 32854 blocking_queue.cpp:50] Data layer prefetch queue empty I0320 18:46:59.492482 32854 solver.cpp:243] Iteration 100, loss = 4.71355 I0320 18:46:59.492624 32854 solver.cpp:259] Train net output #0: mbox_loss = 4.7507 ( 1 = 4.7507 loss) I0320 18:47:18.494647 32854 sgd_solver.cpp:138] Iteration 100, lr = 0.0001 I0320 19:09:15.981941 32854 solver.cpp:243] Iteration 200, loss = 4.25057 I0320 19:09:15.982163 32854 solver.cpp:259] Train net output #0: mbox_loss = 4.12665 ( 1 = 4.12665 loss) I0320 19:09:35.695390 32854 sgd_solver.cpp:138] Iteration 200, lr = 0.0001 I0320 19:28:36.370605 32854 solver.cpp:243] Iteration 300, loss = 4.06544 I0320 19:28:36.370846 32854 solver.cpp:259] Train net output #0: mbox_loss = 3.81313 ( 1 = 3.81313 loss) I0320 19:28:40.893465 32854 sgd_solver.cpp:138] Iteration 300, lr = 0.0001 I0320 19:45:51.642319 32854 solver.cpp:243] Iteration 400, loss = 4.00686 I0320 19:45:51.657042 32854 solver.cpp:259] Train net output #0: mbox_loss = 3.86898 ( 1 = 3.86898 loss) I0320 19:45:52.827033 32854 sgd_solver.cpp:138] Iteration 400, lr = 0.0001 I0320 20:09:07.978211 32854 solver.cpp:243] Iteration 500, loss = 4.04271 I0320 20:09:08.014988 32854 solver.cpp:259] Train net output #0: mbox_loss = 3.87323 ( 1 = 3.87323 loss) I0320 20:09:08.015076 32854 sgd_solver.cpp:138] Iteration 500, lr = 0.0001 I0320 20:40:22.733320 32854 solver.cpp:243] Iteration 600, loss = 3.8724 I0320 20:40:22.733603 32854 solver.cpp:259] Train net output #0: mbox_loss = 3.91798 ( 1 = 3.91798 loss) I0320 20:40:22.733654 32854 sgd_solver.cpp:138] Iteration 600, lr = 0.0001 I0320 21:14:05.252872 32854 solver.cpp:243] Iteration 700, loss = 3.91969 I0320 21:14:05.254623 32854 solver.cpp:259] Train net output #0: mbox_loss = 3.73075 ( 1 = 3.73075 loss) I0320 21:14:12.680538 32854 sgd_solver.cpp:138] Iteration 700, lr = 0.0001 I0320 21:49:44.001958 32854 solver.cpp:243] Iteration 800, loss = 3.86093 I0320 21:49:44.002271 32854 solver.cpp:259] Train net output #0: mbox_loss = 4.65883 ( 1 = 4.65883 loss) I0320 21:49:49.958609 32854 sgd_solver.cpp:138] Iteration 800, lr = 0.0001 I0320 21:53:01.317153 33124 blocking_queue.cpp:50] Data layer prefetch queue empty I0320 22:23:29.791314 32854 solver.cpp:243] Iteration 900, loss = 3.8187 I0320 22:23:29.791862 32854 solver.cpp:259] Train net output #0: mbox_loss = 3.85123 ( 1 = 3.85123 loss) I0320 22:23:47.945513 32854 sgd_solver.cpp:138] Iteration 900, lr = 0.0001 I0320 22:55:44.925760 32854 solver.cpp:243] Iteration 1000, loss = 3.7425 I0320 22:55:44.926020 32854 solver.cpp:259] Train net output #0: mbox_loss = 4.01863 ( 1 = 4.01863 loss) I0320 22:55:50.410193 32854 sgd_solver.cpp:138] Iteration 1000, lr = 0.0001 I0320 23:28:20.539289 32854 solver.cpp:243] Iteration 1100, loss = 3.8035 I0320 23:28:20.548061 32854 solver.cpp:259] Train net output #0: mbox_loss = 3.7838 ( 1 = 3.7838 loss) I0320 23:28:25.631153 32854 sgd_solver.cpp:138] Iteration 1100, lr = 0.0001 I0320 23:56:49.150030 32854 solver.cpp:243] Iteration 1200, loss = 3.72246 I0320 23:56:49.150259 32854 solver.cpp:259] Train net output #0: mbox_loss = 3.75223 ( 1 = 3.75223 loss) I0320 23:57:11.602012 32854 sgd_solver.cpp:138] Iteration 1200, lr = 0.0001 I0321 00:26:45.537853 32854 solver.cpp:243] Iteration 1300, loss = 3.72136 I0321 00:26:45.538089 32854 solver.cpp:259] Train net output #0: mbox_loss = 3.38236 ( 1 = 3.38236 loss) I0321 00:26:45.538132 32854 sgd_solver.cpp:138] Iteration 1300, lr = 0.0001 I0321 00:55:09.132469 32854 solver.cpp:243] Iteration 1400, loss = 3.6804 I0321 00:55:09.132679 32854 solver.cpp:259] Train net output #0: mbox_loss = 2.87209 ( 1 = 2.87209 loss) I0321 00:55:16.543397 32854 sgd_solver.cpp:138] Iteration 1400, lr = 0.0001 I0321 01:31:36.330186 32854 solver.cpp:243] Iteration 1500, loss = 3.64987 I0321 01:31:36.337105 32854 solver.cpp:259] Train net output #0: mbox_loss = 4.11438 ( 1 = 4.11438 loss) I0321 01:32:01.195981 32854 sgd_solver.cpp:138] Iteration 1500, lr = 0.0001 I0321 02:03:18.576989 32854 solver.cpp:243] Iteration 1600, loss = 3.61781 I0321 02:03:18.622884 32854 solver.cpp:259] Train net output #0: mbox_loss = 3.40459 ( 1 = 3.40459 loss) I0321 02:03:18.634691 32854 sgd_solver.cpp:138] Iteration 1600, lr = 0.0001 I0321 02:21:28.850323 32854 blocking_queue.cpp:50] Data layer prefetch queue empty I0321 02:31:13.534430 32854 solver.cpp:243] Iteration 1700, loss = 3.64082 I0321 02:31:13.539944 32854 solver.cpp:259] Train net output #0: mbox_loss = 3.20621 ( 1 = 3.20621 loss) I0321 02:31:35.534457 32854 sgd_solver.cpp:138] Iteration 1700, lr = 0.0001 I0321 03:02:24.651046 32854 solver.cpp:243] Iteration 1800, loss = 3.59635 I0321 03:02:24.651306 32854 solver.cpp:259] Train net output #0: mbox_loss = 3.84726 ( 1 = 3.84726 loss) I0321 03:02:35.240459 32854 sgd_solver.cpp:138] Iteration 1800, lr = 0.0001 I0321 03:28:55.246892 32854 solver.cpp:243] Iteration 1900, loss = 3.64389 I0321 03:28:55.247161 32854 solver.cpp:259] Train net output #0: mbox_loss = 4.57005 ( 1 = 4.57005 loss) I0321 03:29:12.756068 32854 sgd_solver.cpp:138] Iteration 1900, lr = 0.0001 I0321 03:56:30.419265 32854 solver.cpp:243] Iteration 2000, loss = 3.64181 I0321 03:56:30.419488 32854 solver.cpp:259] Train net output #0: mbox_loss = 3.22497 ( 1 = 3.22497 loss) I0321 03:56:30.419528 32854 sgd_solver.cpp:138] Iteration 2000, lr = 0.0001 I0321 04:22:28.350476 32854 solver.cpp:243] Iteration 2100, loss = 3.6115 I0321 04:22:28.356010 32854 solver.cpp:259] Train net output #0: mbox_loss = 4.16131 ( 1 = 4.16131 loss) I0321 04:22:28.356235 32854 sgd_solver.cpp:138] Iteration 2100, lr = 0.0001 I0321 04:48:44.036162 32854 solver.cpp:243] Iteration 2200, loss = 3.56413 I0321 04:48:44.036393 32854 solver.cpp:259] Train net output #0: mbox_loss = 2.78308 ( 1 = 2.78308 loss) I0321 04:48:44.036433 32854 sgd_solver.cpp:138] Iteration 2200, lr = 0.0001 I0321 05:14:25.236363 32854 solver.cpp:243] Iteration 2300, loss = 3.6338 I0321 05:14:25.236584 32854 solver.cpp:259] Train net output #0: mbox_loss = 3.5336 ( 1 = 3.5336 loss) I0321 05:14:28.135422 32854 sgd_solver.cpp:138] Iteration 2300, lr = 0.0001 I0321 05:39:43.394588 32854 solver.cpp:243] Iteration 2400, loss = 3.59064 I0321 05:39:43.394793 32854 solver.cpp:259] Train net output #0: mbox_loss = 3.53245 ( 1 = 3.53245 loss) I0321 05:39:45.348371 32854 sgd_solver.cpp:138] Iteration 2400, lr = 0.0001 I0321 06:02:20.365394 33124 blocking_queue.cpp:50] Data layer prefetch queue empty I0321 06:08:23.288305 32854 solver.cpp:243] Iteration 2500, loss = 3.58889 I0321 06:08:23.289111 32854 solver.cpp:259] Train net output #0: mbox_loss = 2.61665 ( 1 = 2.61665 loss) I0321 06:08:28.295343 32854 sgd_solver.cpp:138] Iteration 2500, lr = 0.0001 I0321 06:43:53.267843 32854 solver.cpp:243] Iteration 2600, loss = 3.47278 I0321 06:43:53.268203 32854 solver.cpp:259] Train net output #0: mbox_loss = 3.52838 ( 1 = 3.52838 loss) I0321 06:44:07.155161 32854 sgd_solver.cpp:138] Iteration 2600, lr = 0.0001 I0321 07:21:03.146634 32854 solver.cpp:243] Iteration 2700, loss = 3.50657 I0321 07:21:03.146906 32854 solver.cpp:259] Train net output #0: mbox_loss = 3.22891 ( 1 = 3.22891 loss) I0321 07:21:30.132973 32854 sgd_solver.cpp:138] Iteration 2700, lr = 0.0001 I0321 07:52:57.703928 32854 solver.cpp:243] Iteration 2800, loss = 3.47005 I0321 07:52:57.704133 32854 solver.cpp:259] Train net output #0: mbox_loss = 3.63058 ( 1 = 3.63058 loss) I0321 07:52:57.704190 32854 sgd_solver.cpp:138] Iteration 2800, lr = 0.0001 I0321 08:19:26.086001 32854 solver.cpp:243] Iteration 2900, loss = 3.49502 I0321 08:19:26.086308 32854 solver.cpp:259] Train net output #0: mbox_loss = 3.85055 ( 1 = 3.85055 loss) I0321 08:19:26.086344 32854 sgd_solver.cpp:138] Iteration 2900, lr = 0.0001 I0321 08:48:30.159132 32854 solver.cpp:243] Iteration 3000, loss = 3.47565 I0321 08:48:30.159410 32854 solver.cpp:259] Train net output #0: mbox_loss = 3.78331 ( 1 = 3.78331 loss) I0321 08:48:30.159449 32854 sgd_solver.cpp:138] Iteration 3000, lr = 0.0001

[wenlihaoyu]

checking the voc xml file whether the "difficult" param all are '1' ?

[justttry]

thanks for your replay @wenlihaoyu . I check the voc xml file and all the 'difficult' param are '0'. I will correct the xml files with '###' labeled '1'.

[dajiangxiaoyan]

@justttry I also met this problem. Do you fix this?

[justttry]

Check the xml files right or not.

The low training speed is just because your training image is too big!! Resize the training images and your training could speed up!! @dajiangxiaoyan

[miaomi1994]

hi,I meet the same problem as you and the loss does not decline ,I trained for 10000 steps but got det_eal=0.33,and then it did't increase.I check the xml.And the program resize the img to 384*384,so I cannot understand what you say about the " training image is too big!!",can you give me some suggestions?Thanks @justttry

[miaomi1994]

hi,I meet the same problem as you and the loss does not decline ,I trained for 10000 steps but got det_eal=0.33,and then it did't increase.I check the xml.Do you solve the problem? @dajiangxiaoyan

[justttry]

the image size is related to the training speed, not the convergence.

check whether the difficultes box is set to 1, and the modelConfig.py is set to pass the difficult boxes.

good luck @miaomi1994