Memory hog problem in `22_deep_reinforcement_learning`/`04_q_learning_for_trading.ipynb`

[DavidHJong]

Describe the bug

• As training goes on, the memory usage by the process increases. It continuously goes up until hit around 10GB in episode 280. In later episodes, memory usage goes up around 70MB per episode, as shown in 04_q_learning_for_trading.ipynb's Train Agent cell's output.

What I did to find this

1. In 04_q_learning_for_trading.ipynb, I added a cell monitor RAM usage with 2 functions to monitor the process memory usage and call these functions multiple times in the Train Agent cell.
2. I recorded the process memory changes and save it into a .csv file and visualize it in a .xlsx file.

Steps to reproduce the behavior:

1. To see in which steps the major memory changes occurred, open the memory_change_record.xlsx file
2. To reproduce this record, run 04_q_learning_for_trading.ipynb.

Questions

• Is this memory leak?
• What is the main part of code contributing to this increasing memory usage?
• Any way to keep the memory usage constant or at a small size as training goes on?

Why it matters

• An increasing memory usage seems like a major reason for the slowing down in later episode training.

[stefan-jansen]

I'm sorry, but I don't have time to investigate. Data leak would be caused by an underlying library, so please take a look at the issues in the relevant libraries. Memory usage could of course increase by keeping certain data around; if that's a problem you could periodically write to disk and reinit the data structures.