search

NVIDIA / caffe

Caffe: a fast open framework for deep learning.
http://caffe.berkeleyvision.org/
Other
672 stars 263 forks source link

Adding few crop layers to an existing model makes it much slower #409

Open mathmanu opened 7 years ago

mathmanu commented 7 years ago

I have an interesting observation. If I add a few Crop layers to the mobilenet model, it becomes quite slow - more than 3x slower. I have a multi GPU setup. Is this happening because Crop layers are not CUDNN accelerated and there are overheads in data movement between GPUs? I can't think of any other reason.

mathmanu commented 7 years ago

Copying the previous conversation from issue #386

mathmanu commented 23 hours ago @drnikolaev I have an interesting observation. If I add a few Crop layers to the mobilenet model, it becomes quite slow - almost 4x slower. I have a multi GPU setup. Is this happening because Crop layers are not CUDNN accelerated and there are overheads in data movement between GPUs? I can't think of any other reason. @RSly

RSly commented 20 hours ago • edited @mathmanu @drnikolaev , I also confirm the problem with the crop layers. in an example: A. with no crop layer, it takes 2hours to train B. with 4 crop layers it takes 3hours to train

that is 1.5x slower... @drnikolaev

drnikolaev commented 11 hours ago @RSly @mathmanu please try this:

transform_param { use_gpu_transform: true ... @mathmanu

mathmanu commented 10 hours ago Setting that parameter for Crop layer had no impact on speed - its still slow. @drnikolaev

drnikolaev commented 10 hours ago @mathmanu actually yes, this setting has nothing to do with CropLayer. Could you please open a request for this? Thank you!