memory usage growth after a while

NVlabs / GA3C

Hybrid CPU/GPU implementation of the A3C algorithm for deep reinforcement learning.

BSD 3-Clause "New" or "Revised" License

649 stars 195 forks source link

memory usage growth after a while #32

Open babak-badnava opened 6 years ago

babak-badnava commented 6 years ago

I have tested the code on GTX 1080 with 32g Ram but when I run the code memory usage increases over time and after about 30 hours it will take all 32g of ram and make system to dies

ifrosio commented 6 years ago

Hi, can you provide some more details? Are you using the basic version of the algorithm with Pong? Automatic scheduling enabled? How many agents / trainers / predictors? Is the number of agents / training / predictor growing during training or does it stabilize? Thanks

babak-badnava commented 6 years ago

Hi, yes I do use the basic version with Breakout game and Dynamic setting is off and 32 agents running with 2 trainer threads and 2 predictor threads. all the settings are fixed during training but memory usage will growth. I think this may be because of tensorflow memory leak.

ifrosio commented 6 years ago

Tensor flow version?

babak-badnava commented 6 years ago

tensorflow version 1.2.1 installing from binary cuda version 7.5