always out of memory after some minute

poojahira / image-captioning-bottom-up-top-down

PyTorch implementation of Image captioning with Bottom-up, Top-down Attention

164 stars 39 forks source link

always out of memory after some minute #10

Closed kangkang59812 closed 4 years ago

kangkang59812 commented 5 years ago

the GPU memory increases along with for-loop when training.

OysterSecret commented 5 years ago

Same issue

poojahira commented 5 years ago

Hi, I was able to successfully train for > 30 epochs using my university GPU cluster with a memory of 120 GB. The code can probably be optimized to use less memory though.

kangkang59812 commented 5 years ago

@poojahira thanks for you reply. I have read all the codes, maybe I should consider more about how to optimal.

poojahira commented 5 years ago

Yes I agree. Please push improvements to this codebase as well if you come across any.

fdgfgdfd commented 4 years ago

loss_d = criterion_dis(scores_d,targets_d.long()) This loss function caused the increasing of the GPU memory while using pytorch-0.4, when I used pytorch-1.1.0, this problem was solved. Maybe it is a bug in pytorch-0.4?