happynear
commented
7 years ago I can tell you one trick. You can see that the softmax loss does not decrease, so close the center loss for several epochs to let softmax loss get down. Then add the center loss back to give some regularization.
Actually I didn't carefully tune the parameter, this model was the first try. You may get better results if you tune the parameters patiently.
Good luck.
I notice that you have trained a center-face model on MS-Celeb-1M dataset and get 99.3% on LFW. However, I do not train such a good results. In fact, my model just can not converge. The loss is around 11.3 all the time. The log file is as follows:
I0622 11:17:24.433667 4178 solver.cpp:279] Solving Face-ResNet I0622 11:17:24.433682 4178 solver.cpp:280] Learning Rate Policy: multistep I0622 11:17:26.033102 4178 solver.cpp:228] Iteration 0, loss = 11.4834 I0622 11:17:26.033166 4178 solver.cpp:244] Train net output #0: center_loss = 18.3962 ( 0.008 = 0.14717 loss) I0622 11:17:26.033192 4178 solver.cpp:244] Train net output #1: softmax_loss = 11.3477 ( 1 = 11.3477 loss) I0622 11:17:26.033219 4178 sgd_solver.cpp:106] Iteration 0, lr = 0.1 I0622 11:17:56.581506 4178 solver.cpp:228] Iteration 20, loss = 11.2866 I0622 11:17:56.581620 4178 solver.cpp:244] Train net output #0: center_loss = 0.827535 ( 0.008 = 0.00662028 loss) I0622 11:17:56.581646 4178 solver.cpp:244] Train net output #1: softmax_loss = 11.2852 ( 1 = 11.2852 loss) I0622 11:17:56.581668 4178 sgd_solver.cpp:106] Iteration 20, lr = 0.1 I0622 11:18:27.520587 4178 solver.cpp:228] Iteration 40, loss = 11.2851 I0622 11:18:27.520741 4178 solver.cpp:244] Train net output #0: center_loss = 0.64857 ( 0.008 = 0.00518856 loss) I0622 11:18:27.520767 4178 solver.cpp:244] Train net output #1: softmax_loss = 11.2808 ( 1 = 11.2808 loss) I0622 11:18:27.520788 4178 sgd_solver.cpp:106] Iteration 40, lr = 0.1 I0622 11:18:59.885659 4178 solver.cpp:228] Iteration 60, loss = 11.28 I0622 11:18:59.885799 4178 solver.cpp:244] Train net output #0: center_loss = 0.602167 ( 0.008 = 0.00481733 loss) I0622 11:18:59.885828 4178 solver.cpp:244] Train net output #1: softmax_loss = 11.2765 ( 1 = 11.2765 loss) I0622 11:18:59.885848 4178 sgd_solver.cpp:106] Iteration 60, lr = 0.1 I0622 11:19:32.551121 4178 solver.cpp:228] Iteration 80, loss = 11.2835 I0622 11:19:32.551265 4178 solver.cpp:244] Train net output #0: center_loss = 0.612441 ( 0.008 = 0.00489953 loss) I0622 11:19:32.551290 4178 solver.cpp:244] Train net output #1: softmax_loss = 11.2802 ( 1 = 11.2802 loss) I0622 11:19:32.551311 4178 sgd_solver.cpp:106] Iteration 80, lr = 0.1 I0622 11:20:05.668840 4178 solver.cpp:228] Iteration 100, loss = 11.2796 I0622 11:20:05.668989 4178 solver.cpp:244] Train net output #0: center_loss = 0.605016 ( 0.008 = 0.00484013 loss) I0622 11:20:05.669018 4178 solver.cpp:244] Train net output #1: softmax_loss = 11.2748 ( 1 = 11.2748 loss) I0622 11:20:05.669039 4178 sgd_solver.cpp:106] Iteration 100, lr = 0.1 I0622 11:20:38.437675 4178 solver.cpp:228] Iteration 120, loss = 11.2818 I0622 11:20:38.437821 4178 solver.cpp:244] Train net output #0: center_loss = 0.573197 ( 0.008 = 0.00458558 loss) I0622 11:20:38.437849 4178 solver.cpp:244] Train net output #1: softmax_loss = 11.2762 ( 1 = 11.2762 loss) I0622 11:20:38.437870 4178 sgd_solver.cpp:106] Iteration 120, lr = 0.1 I0622 11:21:11.199671 4178 solver.cpp:228] Iteration 140, loss = 11.2836 I0622 11:21:11.199820 4178 solver.cpp:244] Train net output #0: center_loss = 0.518494 ( 0.008 = 0.00414796 loss) I0622 11:21:11.199847 4178 solver.cpp:244] Train net output #1: softmax_loss = 11.279 ( 1 = 11.279 loss) I0622 11:21:11.199868 4178 sgd_solver.cpp:106] Iteration 140, lr = 0.1 I0622 11:21:44.417965 4178 solver.cpp:228] Iteration 160, loss = 11.284 I0622 11:21:44.418953 4178 solver.cpp:244] Train net output #0: center_loss = 0.513856 ( 0.008 = 0.00411085 loss) I0622 11:21:44.418978 4178 solver.cpp:244] Train net output #1: softmax_loss = 11.2806 ( 1 = 11.2806 loss) I0622 11:21:44.419000 4178 sgd_solver.cpp:106] Iteration 160, lr = 0.1 I0622 11:22:17.514333 4178 solver.cpp:228] Iteration 180, loss = 11.2805 I0622 11:22:17.514508 4178 solver.cpp:244] Train net output #0: center_loss = 0.48841 ( 0.008 = 0.00390728 loss) I0622 11:22:17.514534 4178 solver.cpp:244] Train net output #1: softmax_loss = 11.2763 ( 1 = 11.2763 loss) I0622 11:22:17.514555 4178 sgd_solver.cpp:106] Iteration 180, lr = 0.1 I0622 11:22:50.642104 4178 solver.cpp:228] Iteration 200, loss = 11.2825 I0622 11:22:50.642247 4178 solver.cpp:244] Train net output #0: center_loss = 0.4449 ( 0.008 = 0.0035592 loss) I0622 11:22:50.642274 4178 solver.cpp:244] Train net output #1: softmax_loss = 11.2771 ( 1 = 11.2771 loss) I0622 11:22:50.642295 4178 sgd_solver.cpp:106] Iteration 200, lr = 0.1 I0622 11:23:23.933121 4178 solver.cpp:228] Iteration 220, loss = 11.2808 I0622 11:23:23.933250 4178 solver.cpp:244] Train net output #0: center_loss = 0.462956 ( 0.008 = 0.00370365 loss) I0622 11:23:23.933276 4178 solver.cpp:244] Train net output #1: softmax_loss = 11.2755 ( 1 = 11.2755 loss) I0622 11:23:23.933298 4178 sgd_solver.cpp:106] Iteration 220, lr = 0.1 I0622 11:23:56.717842 4178 solver.cpp:228] Iteration 240, loss = 11.2803 I0622 11:23:56.717991 4178 solver.cpp:244] Train net output #0: center_loss = 0.428886 ( 0.008 = 0.00343109 loss) I0622 11:23:56.718016 4178 solver.cpp:244] Train net output #1: softmax_loss = 11.2761 ( 1 = 11.2761 loss)
I also try to decrease initial lr to 0.01 or 0.001. It works similarly to 0.1. But I succeed to train my model on CASIA dataset. So could you show me your config file when training MS-Celeb-1M. Is there anything to notice?
Best!