search

MhLiao / TextBoxes_plusplus

TextBoxes++: A Single-Shot Oriented Scene Text Detector
Other
954 stars 279 forks source link

Training is very slow #104

Open 425183525 opened 5 years ago

425183525 commented 5 years ago

Hi, my training speed is very slow. In most cases, 100 iterations will take nearly two hours.I used a single GPU（NVIDIA 1080）, CUDA8.0, cudn 6.0。Tell me, what can I do to improve my training speed。 The image size is 1280x720（1600 images）

image

425183525 commented 5 years ago

@MhLiao

MhLiao commented 5 years ago

You can check the utilization of your GPU. There is a bottleneck in the data loader when the original images are large. But 1280x720 should be OK.

MhLiao commented 5 years ago

The original SSD repo also suffers such problems. It seems that the training speed is sensitive to GPU and drivers. See https://github.com/weiliu89/caffe/issues/691 I guess the low gpu-util is caused by the data layer. You can open the debug mode and see the time consuming of each layer.