RL training stage,out of memory

aimagelab / meshed-memory-transformer

Meshed-Memory Transformer for Image Captioning. CVPR 2020

BSD 3-Clause "New" or "Revised" License

518 stars 136 forks source link

RL training stage,out of memory #55

Closed TBI805 closed 3 years ago

TBI805 commented 3 years ago

Switching to RL, after every epoch training, the memory usage will increase. Finally, the system will be out of memory, and kill process. Could you please give me some tips to solve it? Thank you very much!