search

NVlabs / PWC-Net

PWC-Net: CNNs for Optical Flow Using Pyramid, Warping, and Cost Volume, CVPR 2018 (Oral)
Other
1.64k stars 355 forks source link

Low training speed using Caffe training code #36

Closed ghost closed 6 years ago

ghost commented 6 years ago

Thanks for sharing the code. It is a really impressive work.

I am trying to repeat the result with the caffe training code. However, I observed the training speed is much slower than the stated one (in table 7 of the paper). I used a single NVIDIA 1080ti with Cuda 9.0 on an Ubuntu 16.04 machine. The training speed on FlyingChair is around 5.5K/h, which means at least 1200/ 5.5 = 218 = around 9 days are needed to train only on the FlyingChair. Is there any tip to speed up the training?

In addition, may I ask the training speed stated in table 7 on the paper means training on FlyingChair only or training on FlyingChair + FlyingThings + Kitti/Sintel?

What is the fine-tuning strategy on Kitti/Sintel, did you also use the S_fine schedule (which means in total the training step is 1200k + 500k + 500k = 2200k)?

deqings commented 6 years ago

Thanks for your interest in our work.

The training is for FlyingChairs only. Are you using a HDD? We used SSD to store the LMDB file, which is much faster than HDD.

You can check the details on fine-tuning in this extended version: Models Matter, So Does Training: An Empirical Study of CNNs for Optical Flow Estimation https://arxiv.org/pdf/1809.05571.pdf

rajat95 commented 6 years ago

Hi, When you refer to learning rate disruption in the extended paper, does that mean you keep learning rate very close to 0 for some iterations before increasing it again?