search

weiliu89 / caffe

Caffe: a fast open framework for deep learning.
http://caffe.berkeleyvision.org/
Other
4.77k stars 1.67k forks source link

New SSD model doesn't converge #383

Open PearlDzzz opened 7 years ago

PearlDzzz commented 7 years ago

@weiliu89 I use the new ssd_pascal.py to generate tain model, and tain on my datatset using the new compiled ceffe.exe. However, it doesn't converge.

And the LOG File is : I0110 10:07:37.019037 12020 caffe.cpp:155] Finetuning from VGG_ILSVRC_16_layers_fc_reduced.caffemodel I0110 10:07:37.019037 12020 caffe.cpp:252] Starting Optimization I0110 10:07:37.019037 12020 solver.cpp:294] Solving VGG_VOC0712_SSD_300x300_train I0110 10:07:37.019037 12020 solver.cpp:295] Learning Rate Policy: multistep I0110 10:07:37.633036 12020 solver.cpp:243] Iteration 0, loss = 16.4453 I0110 10:07:37.633036 12020 solver.cpp:259] Train net output #0: mbox_loss = 16.4453 ( 1 = 16.4453 loss) I0110 10:07:37.633036 12020 sgd_solver.cpp:138] Iteration 0, lr = 0.001 I0110 10:07:39.395036 12020 blocking_queue.cpp:50] Data layer prefetch queue empty I0110 10:07:41.492036 12020 solver.cpp:243] Iteration 10, loss = 14.1076 I0110 10:07:41.492036 12020 solver.cpp:259] Train net output #0: mbox_loss = 18.2431 ( 1 = 18.2431 loss) I0110 10:07:41.492036 12020 sgd_solver.cpp:138] Iteration 10, lr = 0.001 I0110 10:07:46.133036 12020 solver.cpp:243] Iteration 20, loss = 16.9603 I0110 10:07:46.133036 12020 solver.cpp:259] Train net output #0: mbox_loss = 18.7958 ( 1 = 18.7958 loss) I0110 10:07:46.133036 12020 sgd_solver.cpp:138] Iteration 20, lr = 0.001 I0110 10:07:51.286036 12020 solver.cpp:243] Iteration 30, loss = 17.0336 I0110 10:07:51.286036 12020 solver.cpp:259] Train net output #0: mbox_loss = 10.1327 ( 1 = 10.1327 loss) I0110 10:07:51.286036 12020 sgd_solver.cpp:138] Iteration 30, lr = 0.001 I0110 10:07:56.052037 12020 solver.cpp:243] Iteration 40, loss = 20.3196 I0110 10:07:56.052037 12020 solver.cpp:259] Train net output #0: mbox_loss = 32.0465 ( 1 = 32.0465 loss) I0110 10:07:56.052037 12020 sgd_solver.cpp:138] Iteration 40, lr = 0.001 I0110 10:08:00.822036 12020 solver.cpp:243] Iteration 50, loss = 18.2571 I0110 10:08:00.822036 12020 solver.cpp:259] Train net output #0: mbox_loss = 17.3493 ( 1 = 17.3493 loss) I0110 10:08:00.822036 12020 sgd_solver.cpp:138] Iteration 50, lr = 0.001 I0110 10:08:05.513036 12020 solver.cpp:243] Iteration 60, loss = 16.2858 I0110 10:08:05.513036 12020 solver.cpp:259] Train net output #0: mbox_loss = 11.9989 ( 1 = 11.9989 loss) I0110 10:08:05.513036 12020 sgd_solver.cpp:138] Iteration 60, lr = 0.001 I0110 10:08:10.280036 12020 solver.cpp:243] Iteration 70, loss = 13.5174 I0110 10:08:10.280036 12020 solver.cpp:259] Train net output #0: mbox_loss = 11.7873 ( 1 = 11.7873 loss) I0110 10:08:10.280036 12020 sgd_solver.cpp:138] Iteration 70, lr = 0.001 I0110 10:08:15.055037 12020 solver.cpp:243] Iteration 80, loss = 13.8097 I0110 10:08:15.055037 12020 solver.cpp:259] Train net output #0: mbox_loss = 20.6231 ( 1 = 20.6231 loss) I0110 10:08:15.055037 12020 sgd_solver.cpp:138] Iteration 80, lr = 0.001 I0110 10:08:19.796036 12020 solver.cpp:243] Iteration 90, loss = 13.569 I0110 10:08:19.796036 12020 solver.cpp:259] Train net output #0: mbox_loss = 15.8057 ( 1 = 15.8057 loss) I0110 10:08:19.796036 12020 sgd_solver.cpp:138] Iteration 90, lr = 0.001 I0110 10:08:24.193037 12020 solver.cpp:433] Iteration 100, Testing net (#0) I0110 10:08:24.459036 12020 net.cpp:693] Ignoring source layer mbox_loss I0110 10:12:45.046036 12020 solver.cpp:546] Test net output #0: detection_eval = 0.0174475 I0110 10:12:45.220036 12020 solver.cpp:243] Iteration 100, loss = 21.9544 I0110 10:12:45.220036 12020 solver.cpp:259] Train net output #0: mbox_loss = 20.4247 ( 1 = 20.4247 loss) I0110 10:12:45.220036 12020 sgd_solver.cpp:138] Iteration 100, lr = 0.001 I0110 10:12:49.376036 12020 solver.cpp:243] Iteration 110, loss = 18.0332 I0110 10:12:49.376036 12020 solver.cpp:259] Train net output #0: mbox_loss = 13.7415 ( 1 = 13.7415 loss) I0110 10:12:49.376036 12020 sgd_solver.cpp:138] Iteration 110, lr = 0.001 I0110 10:12:53.943037 12020 solver.cpp:243] Iteration 120, loss = 18.1878 I0110 10:12:53.943037 12020 solver.cpp:259] Train net output #0: mbox_loss = 8.64498 ( 1 = 8.64498 loss) I0110 10:12:53.943037 12020 sgd_solver.cpp:138] Iteration 120, lr = 0.001 I0110 10:12:58.462036 12020 solver.cpp:243] Iteration 130, loss = 18.0054 I0110 10:12:58.462036 12020 solver.cpp:259] Train net output #0: mbox_loss = 15.9822 ( 1 = 15.9822 loss) I0110 10:12:58.462036 12020 sgd_solver.cpp:138] Iteration 130, lr = 0.001 I0110 10:13:02.602036 12020 solver.cpp:243] Iteration 140, loss = 13.2967 I0110 10:13:02.602036 12020 solver.cpp:259] Train net output #0: mbox_loss = 9.57692 ( 1 = 9.57692 loss) I0110 10:13:02.602036 12020 sgd_solver.cpp:138] Iteration 140, lr = 0.001 I0110 10:13:07.087036 12020 solver.cpp:243] Iteration 150, loss = 12.0768 I0110 10:13:07.087036 12020 solver.cpp:259] Train net output #0: mbox_loss = 7.1722 ( 1 = 7.1722 loss) I0110 10:13:07.088037 12020 sgd_solver.cpp:138] Iteration 150, lr = 0.001 I0110 10:13:11.223037 12020 solver.cpp:243] Iteration 160, loss = 8.4567 I0110 10:13:11.223037 12020 solver.cpp:259] Train net output #0: mbox_loss = 9.30244 ( 1 = 9.30244 loss) I0110 10:13:11.223037 12020 sgd_solver.cpp:138] Iteration 160, lr = 0.001 I0110 10:13:16.025037 12020 solver.cpp:243] Iteration 170, loss = 12.6666 I0110 10:13:16.025037 12020 solver.cpp:259] Train net output #0: mbox_loss = 19.3194 ( 1 = 19.3194 loss) I0110 10:13:16.025037 12020 sgd_solver.cpp:138] Iteration 170, lr = 0.001 I0110 10:13:20.402036 12020 solver.cpp:243] Iteration 180, loss = 15.6799 I0110 10:13:20.402036 12020 solver.cpp:259] Train net output #0: mbox_loss = 26.0719 ( 1 = 26.0719 loss) I0110 10:13:20.402036 12020 sgd_solver.cpp:138] Iteration 180, lr = 0.001 I0110 10:13:25.412036 12020 solver.cpp:243] Iteration 190, loss = 14.774 I0110 10:13:25.413036 12020 solver.cpp:259] Train net output #0: mbox_loss = 10.5424 ( 1 = 10.5424 loss) I0110 10:13:25.413036 12020 sgd_solver.cpp:138] Iteration 190, lr = 0.001 I0110 10:13:29.965036 12020 solver.cpp:433] Iteration 200, Testing net (#0) I0110 10:13:29.965036 12020 net.cpp:693] Ignoring source layer mbox_loss I0110 10:17:38.227036 12020 solver.cpp:546] Test net output #0: detection_eval = 0.0157478 I0110 10:17:38.513036 12020 solver.cpp:243] Iteration 200, loss = 12.1152 I0110 10:17:38.513036 12020 solver.cpp:259] Train net output #0: mbox_loss = 9.1294 ( 1 = 9.1294 loss) I0110 10:17:38.513036 12020 sgd_solver.cpp:138] Iteration 200, lr = 0.001 I0110 10:17:43.265036 12020 solver.cpp:243] Iteration 210, loss = 13.4585 I0110 10:17:43.265036 12020 solver.cpp:259] Train net output #0: mbox_loss = 7.60361 ( 1 = 7.60361 loss) I0110 10:17:43.265036 12020 sgd_solver.cpp:138] Iteration 210, lr = 0.001 I0110 10:17:48.051036 12020 solver.cpp:243] Iteration 220, loss = 14.3801 I0110 10:17:48.051036 12020 solver.cpp:259] Train net output #0: mbox_loss = 11.5277 ( 1 = 11.5277 loss) I0110 10:17:48.051036 12020 sgd_solver.cpp:138] Iteration 220, lr = 0.001 I0110 10:17:52.827036 12020 solver.cpp:243] Iteration 230, loss = 16.3444 I0110 10:17:52.827036 12020 solver.cpp:259] Train net output #0: mbox_loss = 14.9786 ( 1 = 14.9786 loss) I0110 10:17:52.827036 12020 sgd_solver.cpp:138] Iteration 230, lr = 0.001 I0110 10:17:57.657037 12020 solver.cpp:243] Iteration 240, loss = 18.3031 I0110 10:17:57.657037 12020 solver.cpp:259] Train net output #0: mbox_loss = 21.9139 ( 1 = 21.9139 loss) I0110 10:17:57.657037 12020 sgd_solver.cpp:138] Iteration 240, lr = 0.001 I0110 10:18:02.642036 12020 solver.cpp:243] Iteration 250, loss = 15.9183 I0110 10:18:02.642036 12020 solver.cpp:259] Train net output #0: mbox_loss = 14.5084 ( 1 = 14.5084 loss) I0110 10:18:02.642036 12020 sgd_solver.cpp:138] Iteration 250, lr = 0.001 I0110 10:18:07.497036 12020 solver.cpp:243] Iteration 260, loss = 11.0062 I0110 10:18:07.497036 12020 solver.cpp:259] Train net output #0: mbox_loss = 13.1984 ( 1 = 13.1984 loss) I0110 10:18:07.497036 12020 sgd_solver.cpp:138] Iteration 260, lr = 0.001 I0110 10:18:12.463037 12020 solver.cpp:243] Iteration 270, loss = 9.16985 I0110 10:18:12.463037 12020 solver.cpp:259] Train net output #0: mbox_loss = 8.13231 ( 1 = 8.13231 loss) I0110 10:18:12.463037 12020 sgd_solver.cpp:138] Iteration 270, lr = 0.001 I0110 10:18:17.320036 12020 solver.cpp:243] Iteration 280, loss = 9.66784 I0110 10:18:17.320036 12020 solver.cpp:259] Train net output #0: mbox_loss = 11.0557 ( 1 = 11.0557 loss) I0110 10:18:17.320036 12020 sgd_solver.cpp:138] Iteration 280, lr = 0.001 I0110 10:18:22.202036 12020 solver.cpp:243] Iteration 290, loss = 12.6033 I0110 10:18:22.202036 12020 solver.cpp:259] Train net output #0: mbox_loss = 5.66128 ( 1 = 5.66128 loss) I0110 10:18:22.202036 12020 sgd_solver.cpp:138] Iteration 290, lr = 0.001 I0110 10:18:26.538036 12020 solver.cpp:433] Iteration 300, Testing net (#0) I0110 10:18:26.538036 12020 net.cpp:693] Ignoring source layer mbox_loss I0110 10:22:03.945036 12020 solver.cpp:546] Test net output #0: detection_eval = 0.0119402 I0110 10:22:04.314036 12020 solver.cpp:243] Iteration 300, loss = 18.6464 I0110 10:22:04.314036 12020 solver.cpp:259] Train net output #0: mbox_loss = 17.807 ( 1 = 17.807 loss) I0110 10:22:04.314036 12020 sgd_solver.cpp:138] Iteration 300, lr = 0.001 I0110 10:22:09.659036 12020 solver.cpp:243] Iteration 310, loss = 13.3221 I0110 10:22:09.659036 12020 solver.cpp:259] Train net output #0: mbox_loss = 15.8586 ( 1 = 15.8586 loss) I0110 10:22:09.659036 12020 sgd_solver.cpp:138] Iteration 310, lr = 0.001 I0110 10:22:15.009037 12020 solver.cpp:243] Iteration 320, loss = 11.7032 I0110 10:22:15.009037 12020 solver.cpp:259] Train net output #0: mbox_loss = 9.31699 ( 1 = 9.31699 loss) I0110 10:22:15.009037 12020 sgd_solver.cpp:138] Iteration 320, lr = 0.001 I0110 10:22:20.598037 12020 solver.cpp:243] Iteration 330, loss = 13.7108 I0110 10:22:20.598037 12020 solver.cpp:259] Train net output #0: mbox_loss = 4.95746 ( 1 = 4.95746 loss) I0110 10:22:20.598037 12020 sgd_solver.cpp:138] Iteration 330, lr = 0.001 I0110 10:22:26.060036 12020 solver.cpp:243] Iteration 340, loss = 13.5702 I0110 10:22:26.060036 12020 solver.cpp:259] Train net output #0: mbox_loss = 16.3653 ( 1 = 16.3653 loss) I0110 10:22:26.060036 12020 sgd_solver.cpp:138] Iteration 340, lr = 0.001 I0110 10:22:31.464036 12020 solver.cpp:243] Iteration 350, loss = 13.6032 I0110 10:22:31.464036 12020 solver.cpp:259] Train net output #0: mbox_loss = 13.3224 ( 1 = 13.3224 loss) I0110 10:22:31.464036 12020 sgd_solver.cpp:138] Iteration 350, lr = 0.001 I0110 10:22:36.559036 12020 solver.cpp:243] Iteration 360, loss = 10.707 I0110 10:22:36.559036 12020 solver.cpp:259] Train net output #0: mbox_loss = 11.8741 ( 1 = 11.8741 loss) I0110 10:22:36.559036 12020 sgd_solver.cpp:138] Iteration 360, lr = 0.001 I0110 10:22:41.426036 12020 solver.cpp:243] Iteration 370, loss = 15.3335 I0110 10:22:41.426036 12020 solver.cpp:259] Train net output #0: mbox_loss = 8.19201 ( 1 = 8.19201 loss) I0110 10:22:41.426036 12020 sgd_solver.cpp:138] Iteration 370, lr = 0.001 I0110 10:22:46.213037 12020 solver.cpp:243] Iteration 380, loss = 15.7941 I0110 10:22:46.213037 12020 solver.cpp:259] Train net output #0: mbox_loss = 16.7363 ( 1 = 16.7363 loss) I0110 10:22:46.213037 12020 sgd_solver.cpp:138] Iteration 380, lr = 0.001 I0110 10:22:51.038036 12020 solver.cpp:243] Iteration 390, loss = 14.99 I0110 10:22:51.038036 12020 solver.cpp:259] Train net output #0: mbox_loss = 17.3059 ( 1 = 17.3059 loss) I0110 10:22:51.038036 12020 sgd_solver.cpp:138] Iteration 390, lr = 0.001 I0110 10:22:55.415036 12020 solver.cpp:433] Iteration 400, Testing net (#0) I0110 10:22:55.415036 12020 net.cpp:693] Ignoring source layer mbox_loss I0110 10:26:30.497036 12020 solver.cpp:546] Test net output #0: detection_eval = 0.00165196 I0110 10:26:30.663036 12020 solver.cpp:243] Iteration 400, loss = 10.339 I0110 10:26:30.663036 12020 solver.cpp:259] Train net output #0: mbox_loss = 12.6037 ( 1 = 12.6037 loss) I0110 10:26:30.663036 12020 sgd_solver.cpp:138] Iteration 400, lr = 0.001 I0110 10:26:35.427037 12020 solver.cpp:243] Iteration 410, loss = 17.7059 I0110 10:26:35.427037 12020 solver.cpp:259] Train net output #0: mbox_loss = 35.5006 ( 1 = 35.5006 loss) I0110 10:26:35.427037 12020 sgd_solver.cpp:138] Iteration 410, lr = 0.001 I0110 10:26:40.198036 12020 solver.cpp:243] Iteration 420, loss = 22.2176 I0110 10:26:40.198036 12020 solver.cpp:259] Train net output #0: mbox_loss = 28.2043 ( 1 = 28.2043 loss) I0110 10:26:40.199036 12020 sgd_solver.cpp:138] Iteration 420, lr = 0.001 I0110 10:26:44.973037 12020 solver.cpp:243] Iteration 430, loss = 15.209 I0110 10:26:44.973037 12020 solver.cpp:259] Train net output #0: mbox_loss = 8.60929 ( 1 = 8.60929 loss) I0110 10:26:44.973037 12020 sgd_solver.cpp:138] Iteration 430, lr = 0.001 I0110 10:26:49.767036 12020 solver.cpp:243] Iteration 440, loss = 14.1761 I0110 10:26:49.767036 12020 solver.cpp:259] Train net output #0: mbox_loss = 13.7959 ( 1 = 13.7959 loss) I0110 10:26:49.767036 12020 sgd_solver.cpp:138] Iteration 440, lr = 0.001 I0110 10:26:54.573036 12020 solver.cpp:243] Iteration 450, loss = 10.7418 I0110 10:26:54.573036 12020 solver.cpp:259] Train net output #0: mbox_loss = 18.8418 ( 1 = 18.8418 loss) I0110 10:26:54.573036 12020 sgd_solver.cpp:138] Iteration 450, lr = 0.001 I0110 10:26:59.344036 12020 solver.cpp:243] Iteration 460, loss = 14.5299 I0110 10:26:59.344036 12020 solver.cpp:259] Train net output #0: mbox_loss = 19.6312 ( 1 = 19.6312 loss) I0110 10:26:59.344036 12020 sgd_solver.cpp:138] Iteration 460, lr = 0.001 I0110 10:27:04.142036 12020 solver.cpp:243] Iteration 470, loss = 16.8818 I0110 10:27:04.142036 12020 solver.cpp:259] Train net output #0: mbox_loss = 12.9001 ( 1 = 12.9001 loss) I0110 10:27:04.142036 12020 sgd_solver.cpp:138] Iteration 470, lr = 0.001 I0110 10:27:08.916036 12020 solver.cpp:243] Iteration 480, loss = 11.9192 I0110 10:27:08.916036 12020 solver.cpp:259] Train net output #0: mbox_loss = 8.14193 ( 1 = 8.14193 loss) I0110 10:27:08.916036 12020 sgd_solver.cpp:138] Iteration 480, lr = 0.001 I0110 10:27:13.695036 12020 solver.cpp:243] Iteration 490, loss = 13.4667 I0110 10:27:13.695036 12020 solver.cpp:259] Train net output #0: mbox_loss = 11.0473 ( 1 = 11.0473 loss) I0110 10:27:13.695036 12020 sgd_solver.cpp:138] Iteration 490, lr = 0.001 I0110 10:27:18.027036 12020 solver.cpp:433] Iteration 500, Testing net (#0) I0110 10:27:18.028036 12020 net.cpp:693] Ignoring source layer mbox_loss I0110 10:31:00.846036 12020 solver.cpp:546] Test net output #0: detection_eval = 0.00751893

jiangwqcooler commented 7 years ago

smaller lr_rate

moyans commented 7 years ago

you can try base_lr *= 0.1