search

llSourcell / Reinforcement_Learning_for_Stock_Prediction

This is the code for "Reinforcement Learning for Stock Prediction" By Siraj Raval on Youtube
638 stars 362 forks source link

Deque bottleneck on training speed #12

Closed colbyham closed 3 years ago

colbyham commented 6 years ago

When memory deque max length is not equal or much greater than the batch size the training seems to be significantly slowed down. I added a constants file and used the same value for batch_size = BATCH_SIZE in train.py and self.memory = deque(maxlen=BATCH_SIZE) in agent.py.

sailxjx commented 3 years ago

The bottleneck is not deque but the fit/predict cost. If you set maxlen=BATCH_SIZE, the expReplay(fit/predict) will never be executed, then the saved model will have a poor performance no matter how many episodes you train.

sailxjx commented 3 years ago

@llSourcell I think there miss a line of code to clear the 'memory' after this line https://github.com/llSourcell/Reinforcement_Learning_for_Stock_Prediction/blob/master/train.py#L50

sailxjx commented 3 years ago

@colby-ham I come to correct my previous statement after reading the original paper of DQN.

the code in train.py is correct, there is no need to clear the samples in the memory after each step, the model will randomly select samples from memory after each time step.

Therefore, the real reason for the poor performance of the code is that it do not take the benefit of batch training and does not vectorize the train data. I rewrote the replay function in the this repo, you'll see faster training on each step.