How to fix the CUDA out of memory error when using Multi-Step Loss Optimization (MSL)?

AntreasAntoniou / HowToTrainYourMAMLPytorch

The original code for the paper "How to train your MAML" along with a replication of the original "Model Agnostic Meta Learning" (MAML) paper in Pytorch.

https://arxiv.org/abs/1810.09502

Other

773 stars 137 forks source link

How to fix the CUDA out of memory error when using Multi-Step Loss Optimization (MSL)? #25

Open machanic opened 4 years ago

machanic commented 4 years ago

Dear author: when I ran your code I found that https://github.com/AntreasAntoniou/HowToTrainYourMAMLPytorch/blob/master/few_shot_learning_system.py#L196 This line will cause the OOM error because each time the gradient of loss backprop w.r.t. the whole network's parameters will be recorded. And such gradient will not be freed util all the mini-batch's multi-tasks loop over.

I use 11Gb 1080Ti to run CIFAR-10 dataset, but it report CUDA out of memory error. How to deal and fix it?

pandeydeep9 commented 3 years ago

Maybe this could help: reduce the batch size or the number of inner loop updates from 5 to say 2 or 3.