search

xuebinqin / U-2-Net

The code for our newly accepted paper in Pattern Recognition 2020: "U^2-Net: Going Deeper with Nested U-Structure for Salient Object Detection."
Apache License 2.0
8.6k stars 1.48k forks source link

GPU memory increases during training #311

Open kenmbkr opened 2 years ago

kenmbkr commented 2 years ago

I am using the refactored U-2-Net and the GPU memory increases every few iterations during training. I noticed in the code that there are function definitions in the forward functions such as here and here. Would they be the possible causes for the increased memory?

EDIT: Apparently the setting torch.backends.cudnn.benchmark = True is causing the memory to increase during training. Turning it off gives more memory at the first few iterations but the memory still increases as time goes by.

xuebinqin commented 2 years ago

You can try our original implementation of our U-2-Net.

On Wed, Jun 15, 2022 at 12:02 AM kenmbkr @.***> wrote:

I am using the refactored U-2-Net https://github.com/xuebinqin/U-2-Net/blob/master/model/u2net_refactor.py and the GPU memory increases every few iterations during training. I noticed in the code that there are function definitions in the forward functions such as here https://github.com/xuebinqin/U-2-Net/blob/ebb340e24c5645cd75af6c255c8ce3b5eefe074f/model/u2net_refactor.py#L48 and here https://github.com/xuebinqin/U-2-Net/blob/ebb340e24c5645cd75af6c255c8ce3b5eefe074f/model/u2net_refactor.py#L90. Would they be the possible causes for the increased memory?

— Reply to this email directly, view it on GitHub https://github.com/xuebinqin/U-2-Net/issues/311, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADSGOROR6WUPYZ2SFS2HMF3VPF5ZDANCNFSM5Y2FKAWA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

-- Xuebin Qin PhD Department of Computing Science University of Alberta, Edmonton, AB, Canada Homepage: https://xuebinqin.github.io/