Open bamos opened 4 years ago
(Sorry accidentally pressed a hotkey and sent that in early, edited)
This is what this line here does: https://github.com/ikostrikov/pytorch-a2c-ppo-acktr-gail/blob/master/a2c_ppo_acktr/model.py#L113
Ah thanks -- that's pretty hidden and also doesn't hold in my case where I replaced that with my own recurrent policy class. Maybe there should be a well-placed comment or doc somewhere saying that recurrent policies should manually reset their hidden states?
The policy is given the last recurrent state from the replay buffer and isn't reset between episode boundaries. In my case I have the number of updates set to the episode length, so I've added
rollouts.recurrent_hidden_states.zero_()
here: https://github.com/ikostrikov/pytorch-a2c-ppo-acktr-gail/blob/master/main.py#L111But in the more general case I think
done
should be looked at to re-initialize the right states.