search

vlfeat / matconvnet

MatConvNet: CNNs for MATLAB
Other
1.4k stars 756 forks source link

GPU - Out of memory on device #958

Open fshamsafar opened 7 years ago

fshamsafar commented 7 years ago

Hi,

I used "imagenet-resnet-152-dag" for fine-tuning to my dataset. However, I got the error as below. I am using two 6G gpus.

Caused by: Error using dagnn.Sum/forward (line 15) Out of memory on device. To view more detail about available memory on the GPU, use 'gpuDevice()'. If the problem persists, reset the GPU by calling 'gpuDevice(1)'.

How can I solve this problem?

Thank you all

hawescm1 commented 7 years ago

Try reducing the size of your mini-batch.

fshamsafar commented 7 years ago

Thank you @hawescm1 for your help! It did worked by reducing the size of batchSize to 32!

I was wondering whether reducing the batchSize to a small value would affect the final result, or is there any other way to solve this issue other than “reducing the batchSize/using a smaller network/utilizing a better system”?

Thanks again.

hawescm1 commented 7 years ago

I attended a seminar, titled "Stochastic Newton Methods for Machine Learning", given by Jorge Nocedal (September 2016) at Johns Hopkins University, where he espoused using smaller mini-batch sizes in stochastic gradient descent - saying that smaller mini-batch sizes tended to realize deep networks that generalized better.

brealisty commented 7 years ago

@hawescm1 i am sorry, could you tell me where to reduce the size?