How to optimize GPU memory while using lfb

facebookresearch / video-long-term-feature-banks

Long-Term Feature Banks for Detailed Video Understanding

Apache License 2.0

373 stars 62 forks source link

How to optimize GPU memory while using lfb #30

Closed banshee1 closed 5 years ago

banshee1 commented 5 years ago

Hi, I'm sorry to submit a new issue again. I successfully gain a result of 22.8 with r50-baseline which is closed to your report, and I wanna train a r50-lfb model with batch_size 16 on a 8 2080Ti machine. The problem is that the code can run at first but will quit by using out of GPU memory in the first 100 iters. I noticed that there might only be about 200MB excess usage of GPU memory. Is there any way to reduce little GPU usage ? Thanks!

chaoyuaw commented 5 years ago

Hi @banshee1, my pleasure!

Interesting that you got 22.8, higher than what is expected (22.2) with the same model weight. I'm curious, are you able to share what you changed?

I'm sorry but I'm not aware of any way to reduce GPU memory consumption, except for reducing the batch size. Are you training r50-lfb 2L or 3L? Their performance are similar, so if you're training 3L, changing that to 2L will reduce GPU memory usage a little bit without losing much.

chaoyuaw commented 5 years ago

Closing it now, but please feel free to reopen if you have additional questions. Thanks!