search

ijkguo / mx-rcnn

Parallel Faster R-CNN implementation with MXNet.
Other
669 stars 292 forks source link

It's slow to train my own data #101

Closed baolinhu closed 5 years ago

baolinhu commented 5 years ago

Training with VOC data sets, 5.5imgs/second

Using your own dataset (converted to VOC format), only 1.7imgs/second

It's all 4 1080Ti, and the parameters are the same.

Ram-Godavarthi commented 5 years ago

It could be of 2 reasons for slow rate, 1 is your input image size.. and second is your backbone network. I tried with vgg16 as backbone, speed was 2.3 samples/sec with resent 50, speed is 2.5 samples/sec But with resnet 18 , speed is 6.2 samples/sec. I am using aws instance p2.large.. single GPU.

ijkguo commented 5 years ago

It is possible if your dataset has much more objects in one image. Speed is slower most likely due to symdata.bbox_overlaps function, where v5.1 release had a Cython extension.

baolinhu commented 5 years ago

@ijkguo You are right.My dataset has much more objects than voc or coco.It means I should use the v5.1 ,not the latest version? Thanks.

ijkguo commented 5 years ago

Unfortunately yes for now. v6 is nothing better than v5, except simpler usage and less code.