princeton-vl / DeepV2D

BSD 3-Clause "New" or "Revised" License
653 stars 93 forks source link

Why the memory keeps increasing during training #48

Open myalos opened 2 years ago

myalos commented 2 years ago

I train nyu at night and find that it is killed the next day morning. I repeat the training and find the memory keep increasing. My total memory is 32G. I use NVIDIA tensorflow r1.15 https://github.com/NVIDIA/tensorflow/tree/r1.15.