search

CVLAB-Unibo / Unsupervised-Adaptation-for-Deep-Stereo

Code for "Unsupervised Adaptation for Deep Stereo" - ICCV17
Apache License 2.0
62 stars 22 forks source link

The model is not convergent. #7

Open TrackingBird opened 6 years ago

TrackingBird commented 6 years ago

When I fine tuning the network with my dataset, The logs are as follows. I think the model is not convergent. The disparity computed by the fine-tuned model is a white image with a little texture. Can you give me some suggestion about how to train the model? Or how to get a real reliable training dataset. Thanks.


I0508 15:53:59.874593  6385 solver.cpp:245]     Train net output #73719: blob65 = 74.7856
I0508 15:53:59.874598  6385 solver.cpp:245]     Train net output #73720: blob65 = 74.7856
I0508 15:53:59.874603  6385 solver.cpp:245]     Train net output #73721: blob65 = 74.7857
I0508 15:53:59.874608  6385 solver.cpp:245]     Train net output #73722: blob65 = 74.7857
I0508 15:53:59.874614  6385 solver.cpp:245]     Train net output #73723: blob65 = 74.7857
I0508 15:53:59.874619  6385 solver.cpp:245]     Train net output #73724: blob65 = 74.7856
I0508 15:53:59.874624  6385 solver.cpp:245]     Train net output #73725: blob65 = 74.7856
I0508 15:53:59.874629  6385 solver.cpp:245]     Train net output #73726: blob65 = 74.7856
I0508 15:53:59.874634  6385 solver.cpp:245]     Train net output #73727: blob65 = 74.7856
I0508 15:53:59.874646  6385 solver.cpp:245]     Train net output #73728: flow_loss1 = 0.0113061 (* 1 = 0.0113061 loss)
I0508 15:53:59.874652  6385 solver.cpp:245]     Train net output #73729: flow_loss2 = 174.297
I0508 15:53:59.874658  6385 solver.cpp:245]     Train net output #73730: flow_loss3 = 170.482
I0508 15:53:59.874665  6385 solver.cpp:245]     Train net output #73731: flow_loss4 = 163.814
I0508 15:53:59.874670  6385 solver.cpp:245]     Train net output #73732: flow_loss5 = 154.755
I0508 15:53:59.874675  6385 solver.cpp:245]     Train net output #73733: flow_loss6 = 141.625
I0508 15:53:59.874682  6385 solver.cpp:245]     Train net output #73734: loss_down = 0.00255099 (* 0.125 = 0.000318874 loss)
I0508 15:53:59.874689  6385 solver.cpp:245]     Train net output #73735: loss_est = 0.00268216 (* 0.125 = 0.00033527 loss)
I0508 15:53:59.874701  6385 solver.cpp:245]     Train net output #73736: loss_up = 0.00255099 (* 0.125 = 0.000318874 loss)
I0508 15:53:59.874707  6385 solver.cpp:245]     Train net output #73737: loss_west = 0.00268216 (* 0.125 = 0.00033527 loss)
I0508 15:53:59.874717  6385 sgd_solver.cpp:106] Iteration 11950, lr = 3.125e-06
I0508 15:55:18.302011  6385 solver.cpp:456] Snapshotting to binary proto file disp_iter_12000.caffemodel
I0508 15:55:18.302299  6385 net.cpp:867] Serializing 137 layers
I0508 15:55:21.718309  6385 sgd_solver.cpp:273] Snapshotting solver state to binary proto file disp_iter_12000.solverstate
I0508 15:55:29.855378  6385 solver.cpp:318] Iteration 12000, loss = 0.00952777
I0508 15:55:29.855406  6385 solver.cpp:323] Optimization Done.
I0508 15:55:29.855428  6385 caffe.cpp:222] Optimization Done.````
AlessioTonioni commented 6 years ago

If you are using our train.prototxt the model is trying to minimize only flow_loss1 (i.e. the higher resolution disparity) so I think that your model is working. If you take a look at train.prototxt for KITTI each flow_loss except the last have the parameter loss_weight=0 meaning that is not considered during loss computation.