search

vlfeat / matconvnet

MatConvNet: CNNs for MATLAB
Other
1.4k stars 753 forks source link

cudnn makes training slower #822

Open netagy opened 7 years ago

netagy commented 7 years ago

Hi, I am using MatConvNet beta23 on Windows 7 x64 with GeForce GTX TITAN. I'm using CUDA toolkit version 7, and cuDNN v5. When I'm compiling without cudnn support, training a batch takes 10.3 seconds. When I'm compiling with cudnn support, training a batch takes 11.5-12.5 seconds. I was expecting an improvement in performance, and in any case I wasn't expecting training to get slower... Any ideas why it happens? How do I know that the cuDNN is working properly? If it matters I'm trying to train an fcn-8s architecture... Would really appreciate your help, Thanks, Neta

albanie commented 7 years ago

Hi @netagy, just as a sanity check - does the same slowdown occur when training the mnist example?

netagy commented 7 years ago

I checked one epoch of the mnist example. Without cudnn takes 45 seconds, with cudnn takes 46 seconds...

lenck commented 7 years ago

Hi, TITAN is quite old GPU, so what may be the case is that CuDNN needs some of the newer compute capabilities to speed up the convolution operator... In that case, the CuDNN may fall back to standard im2col implementation of convolution, which is also implemented in MatConvNet when compiled without CuDNN support. It is really hard to give anything more than these speculations, as CuDNN is a closed-source library... But what you may try is to install newer CUDA?