search

philtabor / Youtube-Code-Repository

Repository for most of the code from my YouTube channel
861 stars 480 forks source link

[Question] Replay memory #2

Closed TobiasMei closed 5 years ago

TobiasMei commented 5 years ago

https://github.com/philtabor/Youtube-Code-Repository/blob/7c0ca4fbc9a793dc036cb4ac1f3d8caf28019b74/ReinforcementLearning/DeepQLearning/dqn_tf.py#L95

At the Tensorflow Deep QLearning 'dqn_tf.py' file you are using a replay memory.
I have a question about using it.

Then I understand it right, the storage is never cleared?
So, the function store_transition saves the states over epochs?
And in Epoch 2 it is possible that a state from Epoch 1 is trained. So it gets never cleared as long as the program is running.

philtabor commented 5 years ago

The storage is overwritten when the memory is maxed out. So if we set the max memory to 10,000 transitions, the 10,001 transition will overwrite the 1st transition.

When we train, we are randomly sampling the memory, so we're getting states from totally different episodes. We do this to break the correlations that occur during the gameplay of a single episode, so that we can explore a much broader swath of parameter space.