Memory Issues (DQN and DDQN)

[Dewald928]

I have implemented the DQN space invader notebook in Google Colab, as well as in jupyter notebooks. Like clockwork it crashes at 12 episodes, due to a memory (RAM) overflow. When looking at the ram use, it is low until episode 12, then suddenly it jumps and overflows.

I have tried running the .py script as well in pycharm. to no avail. I have reduced the size of the batches and the memory size. I have 16GB ram, which is okay?

Anyone else having this?

Thanks

[KeKeJin]

Hi,

I don't know if you have figured this out already, but I was following the notebook for DDDQN doom, and I had the same memory overflow. To answer your question, I think 16GB ram should be sufficient for the whole project. You shouldn't have used all of your memory at the beginning of your training. Could it be possible that you were storing too much unnecessary data? It might not be relevant to you, but here is what I found out about my memory overflow. I had a Memory object to store the experience, and before training, I attempted to pre-populate the Memory object with 100000 random experience, but I ran out of RAM at like ~100th storing in the Memory object. In my case, one experience includes current_state, action, reward, next_state, done(bool). It turned out that both the current_state and next_state were lists of 3 identical elements, so I replaced each list with one of its element, and it worked well for me.

[Dewald928]

Can you maybe post the parts of the code you changed here?

Thanks so much for the effort.

[KeKeJin]

Of course! To get the element from the lists of three elements, I did this: state = game.get_state().screen_buffer[0] I also made some other changes for the code to run, however, even though my code runs, there should be other problems and I am still figuring out. For instance, my game.is_episode_finished() always returns False, so the game never ends.

1. Prerequisites in the DDDQN notebook: to help install vizdoom !apt-get install build-essential zlib1g-dev libsdl2-dev libjpeg-dev \ nasm tar libbz2-dev libgtk2.0-dev cmake git libfluidsynth-dev libgme-dev \ libopenal-dev timidity libwildmidi-dev unzip !apt-get install libboost-all-dev !apt-get install liblua5.1-dev !pip install vizdoom This takes around 2~5 minutes
2. Step 2 in the DDDQN notebook, to upload the configuration files, I first download @simoninithomas 's config files (deadly_corridor.cfg, deadly_corridor.wad, and deadly_corridor_testing.cfg), and run this code: from google.colab import files file = files.upload() to upload these files to Colab also, in the method create_environment(), before game.init(), add this game.set_window_visible(False), because I don't know how to visualize the training in game view in Colab. Maybe there's an another way around this.
3. Step 4 in the DDDQN notebook:action_size = game.get_available_buttons_size() gives me 0, so I manually set the action_size to 7; pretrain_length = 100000 pretrain_length = 100000 this takes too much memory, so I change it to pretrain_length = 50000 pretrain_length = 50000
4. Step 6 and Step 7 in the DDDQN notebook I changed every occurrence of game.get_state().screen_buffer to be game.get_state().screen_buffer[0], as mentioned above 5.Step 7 in the DDDQN notebook At the beginning of the training block, instead of if training == True: with tf.Session() as sess: I did if training == False: with tf.Session() as sess: otherwise the whole code cell doesn't really do anything I also commented on the write TF Summaries part. I don't know how to write tensorBoard summary in Colab.