search

openai / atari-py

A packaged and slightly-modified version of https://github.com/bbitmaster/ale_python_interface
GNU General Public License v2.0
373 stars 183 forks source link

Random seed parameter #40

Closed marintoro closed 5 years ago

marintoro commented 5 years ago

When I change the parameter random_seed (I am using atari-py 0.1.1), atari_py.ALEInterface().setInt('random_seed', new_seed), it actually doesn't change anything (at least on game space_invaders), my agent without any randomness (I set repeat_action_probability to 0) will always reach the same score over different random_seed. I would expect the Alien to shoot at different time and then lead to different runs, am I wrong?

AdrienLE commented 5 years ago

There are several ways in which Atari games generate random numbers. Many are in fact 100% deterministic, and any random-seeming behavior is just a result of the fact that humans are never consistent in their behavior (which doesn't happen when replaying actions). Others use the user's actions as a source of randomness, which again means that replaying actions should give the same results, and a small minority (Solaris is the only one I'm aware of) do in fact use the randomness provided by .seed() which, based on my reverse-engineering, seems to mainly (or entirely) consist in different random initializations of the RAM and registers at startup.

In short, only a small number of games should be impacted by .seed() simply due to the fact that the sequence of user actions is the primary source of stochasticity in Atari, so it wouldn't be surprising/buggy if .seed had no effect in space invaders.

References: This paper, which (incorrectly) states that user actions are the only source of stochasticity in Atari, showing just how dominant this approach to randomness is in Atari games: https://www.cs.utexas.edu/~pstone/Papers/bib2html/b2hd-AAAI15-hausknecht.html

christopherhesse commented 5 years ago

Thanks for answering this @AdrienLE!