search

weiliu89 / caffe

Caffe: a fast open framework for deep learning.
http://caffe.berkeleyvision.org/
Other
4.77k stars 1.68k forks source link

Big loss after 200K iters training using squeeze+resnet body #135

Open guo253 opened 8 years ago

guo253 commented 8 years ago

Hi, when i used resnet101 for traning(just using the code downloaded) ,it had not gotten lower loss after long time,because there is only 1 GPU. So,I changed the body to squeeze+resnet,and altered "def AddExtraLayers" and "mbox_source_layers",the other code was not changed. But there was big loss after 200K iters training: I0822 15:16:48.064829 9937 sgd_solver.cpp:106] Iteration 119980, lr = 5e-10 I0822 15:16:50.254401 9937 solver.cpp:231] Iteration 119990, loss = 4.01611 I0822 15:16:50.254508 9937 solver.cpp:247] Train net output #0: mbox_loss_gr = 3.7169 (* 1 = 3.7169 loss) ... I0822 15:16:52.715543 9937 solver.cpp:320] Iteration 120000, loss = 4.09774 I0822 15:16:52.715570 9937 solver.cpp:421] Iteration 120000, Testing net (#0) I0822 15:16:52.715649 9937 net.cpp:684] Ignoring source layer mbox_loss_gr I0822 15:17:46.646358 9937 solver.cpp:531] Test net output #0: detection_eval = 0.292901

Do you have any suggestions? lr was small enough from 10^-6 step to 10^-10. Thanks.

weiliu89 commented 8 years ago

Your learning rate seems to be too small. Did you try using the default learning rate from the script?

guo253 commented 8 years ago

I have tried the base lr,but the loss was "Iteration 82240, loss = 5.10802" for a long time,even more than 100K iters. I just altered the "mbox_source_layers,addextralayers,Squeeze_Resnet_Body "in ssd_pascal.py ,and the code in model_libs.py. The num of stride=2 is same as VGGBody which works correctly. When the body is needed to change,which params are needed to care about? I want to cut the size of the total params. Thank you very much.

guo253 commented 8 years ago

I have reached 69% for Squeeze+ResNet Body for about 50K iters,just add BN for 'CreateMultiBoxHead'. Next,I want to cut size more and hope the acc is higher. Beg for your suggestion. Thanks for your work.

hengck23 commented 8 years ago

@guo253 squeeze+resnet is very small and I wonder what is the speed you get for it? How is it compare to vgg16 SSD?