search

weiliu89 / caffe

Caffe: a fast open framework for deep learning.
http://caffe.berkeleyvision.org/
Other
4.77k stars 1.68k forks source link

Training on larger images #562

Open raviv opened 7 years ago

raviv commented 7 years ago

Hi,

My video input is 1800x1200 and so far I've been successfully using SSD 512 by resizing the input at training and test, but was wondering if I would get better results by modifying the net, to support this resolution and avoid resizing. I realize it would make everything slower but accuracy is more important than speed in my case. Any advice on wether this would affect results and pointers to required changes would be much appreciated.

Thanks, Raviv

weiliu89 commented 7 years ago

You could try to fine-tune a bigger model based on the 512 model.

raviv commented 7 years ago

Thanks @weiliu89. I assume I would have to change these as well? https://github.com/weiliu89/caffe/blob/ssd/examples/ssd/ssd_pascal.py#L86 https://github.com/weiliu89/caffe/blob/ssd/examples/ssd/ssd_pascal.py#L299 https://github.com/weiliu89/caffe/blob/ssd/examples/ssd/ssd_pascal.py#L318

OMG59E commented 7 years ago

i trained my own data with 2*1080 SLI, but train`s speed is very slow. why?

I0511 19:26:57.251538 18564 sgd_solver.cpp:138] Iteration 940, lr = 0.001 I0511 19:46:44.235342 18564 solver.cpp:243] Iteration 950, loss = 2.6802 I0511 19:46:44.235447 18564 solver.cpp:259] Train net output #0: mbox_loss = 2.96073 ( 1 = 2.96073 loss) I0511 19:46:44.235482 18564 sgd_solver.cpp:138] Iteration 950, lr = 0.001 I0511 20:05:00.487332 18564 solver.cpp:243] Iteration 960, loss = 2.6615 I0511 20:05:00.487445 18564 solver.cpp:259] Train net output #0: mbox_loss = 2.77214 ( 1 = 2.77214 loss) I0511 20:05:00.487484 18564 sgd_solver.cpp:138] Iteration 960, lr = 0.001 I0511 20:26:18.896340 18564 solver.cpp:243] Iteration 970, loss = 2.87716 I0511 20:26:18.896518 18564 solver.cpp:259] Train net output #0: mbox_loss = 3.5869 ( 1 = 3.5869 loss) I0511 20:26:18.896608 18564 sgd_solver.cpp:138] Iteration 970, lr = 0.001 I0511 20:56:44.085966 18564 solver.cpp:243] Iteration 980, loss = 2.71481 I0511 20:56:44.088560 18564 solver.cpp:259] Train net output #0: mbox_loss = 3.1598 ( 1 = 3.1598 loss) I0511 20:57:22.874410 18564 sgd_solver.cpp:138] Iteration 980, lr = 0.001 I0511 21:19:43.168467 18564 solver.cpp:243] Iteration 990, loss = 2.71404 I0511 21:19:43.168947 18564 solver.cpp:259] Train net output #0: mbox_loss = 2.96801 ( 1 = 2.96801 loss) I0511 21:22:22.475548 18564 sgd_solver.cpp:138] Iteration 990, lr = 0.001 I0511 21:50:04.035192 18564 solver.cpp:243] Iteration 1000, loss = 2.72257 I0511 21:50:04.040537 18564 solver.cpp:259] Train net output #0: mbox_loss = 2.41076 ( 1 = 2.41076 loss) I0511 21:51:49.709125 18564 sgd_solver.cpp:138] Iteration 1000, lr = 0.001

weiliu89 commented 7 years ago

Probably the data augmentation is taking to long to complete. There are ways to optimize the speed.