search

Kaixhin / Rainbow

Rainbow: Combining Improvements in Deep Reinforcement Learning
MIT License
1.56k stars 282 forks source link

Data Effiecient Rainbow with Skiing does not work #86

Closed Somjit77 closed 1 year ago

Somjit77 commented 1 year ago

There is a problem with sampling from the replay buffer with data-efficient rainbow hyper-parameters with 'skiing'. It goes into an infinite loop.

Kaixhin commented 1 year ago

Checking the paper, it seems that the algorithm was only tested on 26 games, which doesn't include "skiing", so there's no guarantee in the original work that those hyperparameters are valid for the entire Atari suite.

Somjit77 commented 1 year ago

Hi, thanks for pointing this out. However, I still do not understand why it would affect prioritized experience replay sampling. If you run the algorithm with skiing for seed=123, it simply goes into an infinite loop while sampling from the buffer. It works perfectly for uniform sampling, so I am unable to figure out why this would be the case.

Kaixhin commented 1 year ago

Looks like a duplicate of https://github.com/Kaixhin/Rainbow/issues/41. PER can get stuck depending on the way its sampling works - as you can see from how uniform sampling works fine.

Somjit77 commented 1 year ago

Ah I see, I think the fact that we use 20-step TD compounds this problem as well. Thank you so much. It makes sense now.