Closed albertwujj closed 5 years ago
Could be why https://github.com/Kaixhin/Rainbow/issues/16
It should be handled by these conditions which make sure that samples near the index are discarded and resampled. Go ahead and work through the logic (I'm not 100% certain that this is bug-free, but I believe it's fine).
Cool! On a first read it looks correct (I will certainly let you know if I discover its not).
I'm modifying your ReplayBuffer to allow parallel envs (still store all transitions in a 1D array, but step back num_envs indices rather than 1 index to get previous timestep). The code you linked is actually a place I missed updating, leading to a bug (getting None when looking for n-step transition states). Thank you so much for the help!
Oops, I see there is a whole paper on 'distributed' replay: https://arxiv.org/pdf/1803.00933.pdf
As segment tree uses a cyclic buffer, the beginning of some transitions (the history part) will be overwritten by new transitions. Thus the 'history' transitions will not be the actual previous transitions. I do not see the code to handle this. Is it a bug?