nczempin
commented
7 years ago On my Kaboom! rom add, I came upon this issue, because the game freezes when you reach the max score (999,999).
Without having seen this issue, I decided that termination was the logical choice.
Now that I think about it, just checking ">= 999999" is not sufficient, because in that game, the maximum reward you can get is 8 (AFAICS from the manual), so I should check "> 999991". However, since this particular game just hangs, I assume it will eventually hit a timeout of sorts (does it? if not, that should at least be some kind of option, perhaps based on the time since the last reward). Right now each game can do that individually, but it is preferred to have a unified method, as you suggest. such a unified method would need the maximum score and the maximum reward passed from each game.
These values can be determined manually, or at least a good first approximation could be automated, by writing successively larger values into the score memory until something happens (and a better-than-nothing first approximation of "highest possible reward" could be obtained by running a few decent agents, perhaps adding a safety margin).
Incidentally, it would be a good idea to generalize the concept of "score memory" and "lives memory", because then many new (and existing) roms could simply be supported by providing those two sets of values as parameters, rather than duplicating all the code that does the reading, extracting reward, etc.
mgbellemare
On some games it is possible to loop the score. This has always been the case but is a greater issue on games where it is possible to learn policies that plays forever, e.g. on Atlantis. There should be a unifying scheme for dealing with score wrapping in games where this can occur. Either:
Note that there are games, e.g. Krull, where a simple agent can loop the score without achieving anything meaningful. This needs to be taken into consideration when evaluating agents that loop the score.