search

ijkguo / mx-rcnn

Parallel Faster R-CNN implementation with MXNet.
Other
669 stars 292 forks source link

multi-gpu problem #63

Closed niluanwudidadi closed 7 years ago

niluanwudidadi commented 7 years ago

@precedenceguo I have met a problem in multi-gpu rcnn training,when i use 4 1080, the speed is no faster than 2 gpu, and the terminal report the message "only 4 out of 12 GPU pairs are enabled direct access.The command is python train_end2end --gpu 0,1,2,3. Is the vgg network's reason?when I use resnet the speed is increases linearly with the number of gpu; and is there some parameters should set?

when i use python train_end2end --gpu 0 ,the speed is 5.4 samples/s when i use python train_end2end --gpu 0,1 ,the speed is 8 samples/s when i use python train_end2end --gpu 0,1,2 ,the speed is 7.4 samples/s when i use python train_end2end --gpu 0,1,2,3 ,the speed is 6.4 samples/s

when i use python train_end2end --network resnet --gpu 0 ,the speed is 2.3 samples/s when i use python train_end2end --network resnet --gpu 0,1 ,the speed is 4.0 samples/s when i use python train_end2end --network resnet --gpu 0,1,2 ,the speed is 5.0 samples/s when i use python train_end2end --network resnet --gpu 0,1,2,3 ,the speed is 5.7 samples/s

ijkguo commented 7 years ago

There is no trivial answer, nor published solutions due to heterogeneous hardware, different environment, etc.

niluanwudidadi commented 7 years ago

Thank you! @precedenceguo