search

BVLC / caffe

Caffe: a fast open framework for deep learning.
http://caffe.berkeleyvision.org/
Other
34.13k stars 18.68k forks source link

memory consumption deconvolution layer with cudnn #6284

Closed pfrenzel closed 6 years ago

pfrenzel commented 6 years ago

Issue summary

The deconvolution layer requires up to three times more memory with cudnn.

Steps to reproduce

I use an hourglass-model which includes some deconvolution layers for upsampling. The parameters are set as described in this documentation.

With the current master branch, the model's memory consumption has increased significantly. As soon as I deactivate the cudnn code of the deconvolution layers (in layer_factory), everything is fine again.

Memory without cudnn for deconvolution layer: ~ 2 GB Memory with cudnn for deconvolution layer: ~ 6 GB

Your system configuration

Operating system: Ubuntu 16.04 Compiler: gcc 5.4 CUDA version (if applicable): 9.1 CUDNN version (if applicable): 7.0.5 BLAS: yes Python or MATLAB version (for pycaffe and matcaffe respectively): N/A

shelhamer commented 6 years ago

cuDNN includes different conv./deconv. algorithms and these vary in their time and space complexity. If you aren't setting group = num_output then you can do so, but otherwise the only choice is to switch between Caffe and cuDNN implementations with the engine setting: https://github.com/BVLC/caffe/blob/master/src/caffe/proto/caffe.proto#L613

Closing since cuDNN is its own library, and so this comes down to usage.