Closed colbyham closed 3 years ago
The bottleneck is not deque
but the fit/predict cost. If you set maxlen=BATCH_SIZE
, the expReplay(fit/predict)
will never be executed, then the saved model will have a poor performance no matter how many episodes you train.
@llSourcell I think there miss a line of code to clear the 'memory' after this line https://github.com/llSourcell/Reinforcement_Learning_for_Stock_Prediction/blob/master/train.py#L50
@colby-ham I come to correct my previous statement after reading the original paper of DQN.
the code in train.py is correct, there is no need to clear the samples in the memory after each step, the model will randomly select samples from memory after each time step.
Therefore, the real reason for the poor performance of the code is that it do not take the benefit of batch training and does not vectorize the train data. I rewrote the replay function in the this repo, you'll see faster training on each step.
When memory deque max length is not equal or much greater than the batch size the training seems to be significantly slowed down. I added a constants file and used the same value for
batch_size = BATCH_SIZE
in train.py andself.memory = deque(maxlen=BATCH_SIZE)
in agent.py.