search

Farama-Foundation / Gymnasium

An API standard for single-agent reinforcement learning environments, with popular reference environments and related utilities (formerly Gym)
https://gymnasium.farama.org
MIT License
6.44k stars 734 forks source link

[Question] How should frame pooling on *-v5 Atari envs be applied? #702

Closed skandermoalla closed 4 months ago

skandermoalla commented 11 months ago

Question

It is not clear how frame pooling (component-wise maximum over two consecutive frames [1,2]) should be applied with the *-v5 version of the Atari environments as they implement frame skipping internally. As far as I understand pooling is generally applied to the last two environment frames (i.e. the last skipped frame and the one after that would be shown to the agent). Without access to the skipped frame, pooling would be applied to the last two frames shown to the agent and would defeat the purpose of capturing alternating sprites [1, 2].

I couldn't find any documentation on this or a relevant wrapper for it. Any clarification is welcome. Thanks!

[1] https://danieltakeshi.github.io/2016/11/25/frame-skipping-and-preprocessing-for-deep-q-networks-on-atari-2600-games/ [2] Machado et al. “Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents” Journal of Artificial Intelligence Research (2018) URL: https://jair.org/index.php/jair/article/view/11182.

pseudo-rnd-thoughts commented 11 months ago

Great question, for some reason I thought that this was done internally.

I found this issue discussing it - https://github.com/Farama-Foundation/Arcade-Learning-Environment/issues/467

There is a open pr with a proposed solution but doesn't look it has been accepted yet

skandermoalla commented 11 months ago

Thanks a lot for the pointer! Should we keep this issue open to track it and then add it to the Gymnasium documentation?

pseudo-rnd-thoughts commented 11 months ago

Yes it is probably a good idea to note this to prevent future users from having the same issue

skandermoalla commented 4 months ago

@pseudo-rnd-thoughts do you have any updates that completed this issue?

pseudo-rnd-thoughts commented 4 months ago

You can specify a frameskip=1 with gym.make("ALE/Pong-v5", frameskip=1) which would allow you to implement the frameskip yourself like the AtariPreprocessing wrapper

skandermoalla commented 4 months ago

Fair enough. I guess this makes this issue independent of https://github.com/Farama-Foundation/Arcade-Learning-Environment/issues/467.