Training takes forever to start

MhLiao / TextBoxes_plusplus

TextBoxes++: A Single-Shot Oriented Scene Text Detector

Other

954 stars 279 forks source link

Training takes forever to start #110

Closed ryanaleksander closed 5 years ago

ryanaleksander commented 5 years ago

Hello. I'm using the docker image to train on my own dataset, but it's been stuck at the same place for hours, it's still stuck right now as I'm typing this. I don't know if it's even running properly or not since it's barely using any GPU memory.

Here are some screenshots

MhLiao commented 5 years ago

@ryanaleksander The original SSD repo also suffers such problems. It seems that the training speed is sensitive to GPU and drivers. See https://github.com/weiliu89/caffe/issues/691 I guess the low gpu-util is caused by the data layer. You can open the debug mode and see the time consuming of each layer.

ryanaleksander commented 5 years ago

Ok, I fixed it. I just happened to use the same dataset for both training and testing so it didn't run