DQN pong example doesn't work off the shelf

[thisiscam]

Describe the bug

Running the DQN example on pong generates the following error when generating a gif:

  File ".../lib/python3.9/site-packages/coax/utils/_misc.py", line 475, in generate_gif
    assert env.render_mode == 'rgb_array', "env.render_mode must be 'rgb_array'"

This is likely due to some recent updates to gym. Currently, on gym==0.26.2 I observe the following:

import gym
env = gym.make('PongNoFrameskip-v4', render_mode="rgb_array")
print(env.render_mode) # prints None

[whaiao]

I was able to isolate this behavior to only Atari environments. I expect something to go wrong while building the environment with their latest patch (https://github.com/openai/gym/blob/6a04d49722724677610e36c1f92908e72f51da0c/gym/envs/registration.py#L631), where the render mode is automatically set to None, but internally stays the same way, because env.render() returns an np.ndarray.

I proposed a small but working change in the above's PR.

– Ben

[KristianHolsheimer]

Thanks for reporting this @thisiscam and thanks so much for digging into this @benjaminbeilharz

Would you mind filing a bug upstream in https://github.com/openai/gym ?

I'll review your PR now.

[whaiao]

@KristianHolsheimer I have filed an issue here: https://github.com/openai/gym/issues/3166

Thanks for your work on coax :)

[KristianHolsheimer]

Fixed by #38