search

MhLiao / TextBoxes_plusplus

TextBoxes++: A Single-Shot Oriented Scene Text Detector
Other
956 stars 281 forks source link

how long will textboxes++ takes for training? #12

Closed codeVerySlow closed 6 years ago

codeVerySlow commented 6 years ago

I0209 09:00:36.055371 113790 blocking_queue.cpp:50] Data layer prefetch queue empty I got this message when start train with rctw-2017 dataset

I train with p100 gpu 10 iteration takes 3 minutes

How long it takes to do 1 iteration when you do training ？ Did I do something wrong? @MhLiao

melgor commented 6 years ago

I'm using V100 with my own dataset and 100 iteration take 20 minutes (so 10 iter take 2 minutes) The main reason for that is only one thread for data preparation, most of time the GPU is not working. Maybe it is because I have 200 BB per image. I will check if data-prefetching could be faster and let you know.

MhLiao commented 6 years ago

@codeVerySlow The image resolution of RCTW-17 is large so the data prefetching and data augmentation are heavy. Thus, the training speed on RCTW-17 is much slower. It can achieve about 1.2 seconds per iter when training on ICDAR 2015 Incidental Text, where the resolution of the images is of 720*1280.