Poor introduction to dopamine

So I tried following the overview and - to be honest - this has been a pain.

The entry point to the standard Atari 2600 experiment is [dopamine/discrete_domains/train.py](https://github.com/google/dopamine/blob/master/dopamine/discrete_domains/train.py). To run the basic DQN agent,

python -um dopamine.discrete_domains.train \
  --base_dir /tmp/dopamine_runs \
  --gin_files dopamine/agents/dqn/configs/dqn.gin

This raises loads of errors:

First all the missing imports (I fixed those using pipreqs).
Then ModuleNotFoundError: No module named 'dopamine.metrics' (described in GH-196) that I fixed by changing all from dopamine.metrics to from metrics.

Finaly I've got Gym complaining it doesn't distribute ROMs anymore:

gym.error.Error: We're Unable to find the game "Pong". Note: Gym no longer distributes ROMs. If you own a license to use the necessary ROMs for research purposes you can download them via `pip install gym[accept-rom-license]`. Otherwise, you should try importing "Pong" via the command `ale-import-roms`. If you believe this is a mistake perhaps your copy of "Pong" is unsupported. To check if this is the case try providing the environment variable `PYTHONWARNINGS=default::ImportWarning:ale_py.roms`. For more information see: https://github.com/mgbellemare/Arcade-Learning-Environment#rom-management

I believe all of this should run onto the core and atari containers that I created following the readme but I don't see where this is explained...

So I started looking at the colabs hoping to find some more infos but I got loads of errors...

agents.ipynb in Load baseline data

---------------------------------------------------------------------------

TypeError                                 Traceback (most recent call last)

[<ipython-input-6-76246b013715>](https://localhost:8080/#) in <module>()
      1 # @title Load baseline data
      2 get_ipython().system('gsutil -q -m cp -R gs://download-dopamine-rl/preprocessed-benchmarks/* /content/')
----> 3 experimental_data = colab_utils.load_baselines('/content')

1 frames

[/usr/lib/python3.7/copyreg.py](https://localhost:8080/#) in _reconstructor(cls, base, state)
     41 def _reconstructor(cls, base, state):
     42     if base is object:
---> 43         obj = object.__new__(cls)
     44     else:
     45         obj = base.__new__(cls, state)

TypeError: object.__new__(BlockManager) is not safe, use BlockManager.__new__()

load_statistics.ipynb in Load the baseline data

---------------------------------------------------------------------------

TypeError                                 Traceback (most recent call last)

<ipython-input-4-fed9ae80a060> in <module>()
      2 
      3 get_ipython().system('gsutil -q -m cp -R gs://download-dopamine-rl/preprocessed-benchmarks/* /content/')
----> 4 experimental_data = colab_utils.load_baselines('/content')

1 frames

/usr/lib/python3.7/copyreg.py in _reconstructor(cls, base, state)
     41 def _reconstructor(cls, base, state):
     42     if base is object:
---> 43         obj = object.__new__(cls)
     44     else:
     45         obj = base.__new__(cls, state)

TypeError: object.__new__(BlockManager) is not safe, use BlockManager.__new__()

agent_visualizer.ipynb in Generate the video

pygame 2.1.2 (SDL 2.0.16, Python 3.7.13)
Hello from the pygame community. https://www.pygame.org/contribute.html
WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/tensorflow/python/compat/v2_compat.py:107: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
Instructions for updating:
non-resource variables are not supported in the long term

---------------------------------------------------------------------------

Exception                                 Traceback (most recent call last)

[<ipython-input-3-05aad0365c1e>](https://localhost:8080/#) in <module>()
      4 example_viz_lib.run(agent='rainbow', game='SpaceInvaders', num_steps=num_steps,
      5                     root_dir='/tmp/agent_viz', restore_ckpt='/tmp/tf_ckpt-199',
----> 6                     use_legacy_checkpoint=True)

15 frames

[/usr/local/lib/python3.7/dist-packages/atari_py/games.py](https://localhost:8080/#) in get_game_path(game_name)
     18     path = os.path.join(_games_dir, game_name) + ".bin"
     19     if not os.path.exists(path):
---> 20         raise Exception('ROM is missing for %s, see https://github.com/openai/atari-py#roms for instructions' % (game_name,))
     21     return path
     22 

Exception: ROM is missing for space_invaders, see https://github.com/openai/atari-py#roms for instructions
  In call to configurable 'create_atari_environment' (<function create_atari_environment at 0x7f4b9399da70>)
  In call to configurable 'Runner' (<class 'dopamine.discrete_domains.run_experiment.Runner'>)

agent_visualizer.ipynb in Generate video

WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/tensorflow/python/compat/v2_compat.py:107: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version.
Instructions for updating:
non-resource variables are not supported in the long term

---------------------------------------------------------------------------

Exception                                 Traceback (most recent call last)

[<ipython-input-5-527301e7cc72>](https://localhost:8080/#) in <module>()
     12 gin.parse_config(config)
     13 runner = example_viz_lib.MyRunner(base_dir, '/tmp/ckpt.199',
---> 14                                   AGENT_CREATORS[agent])
     15 runner.visualize(base_dir / 'images', num_global_steps=200)

13 frames

[/usr/local/lib/python3.7/dist-packages/atari_py/games.py](https://localhost:8080/#) in get_game_path(game_name)
     18     path = os.path.join(_games_dir, game_name) + ".bin"
     19     if not os.path.exists(path):
---> 20         raise Exception('ROM is missing for %s, see https://github.com/openai/atari-py#roms for instructions' % (game_name,))
     21     return path
     22 

Exception: ROM is missing for breakout, see https://github.com/openai/atari-py#roms for instructions
  In call to configurable 'create_atari_environment' (<function create_atari_environment at 0x7f9a4a7ff200>)
  In call to configurable 'Runner' (<class 'dopamine.discrete_domains.run_experiment.Runner'>)

happily tensorboard.ipynb functions properly and cartpole.ipynb too.

Overall, all of this makes for a very poor introduction to dopamine.

sorry about this! we made some changes to the dopamine API recently, which probably broke a few things. i'll look into this next week and have it resolved quickly.

apologies for the inconvenience, but thanks for raising it!

cheers,

-psc

On Thu, May 19, 2022 at 9:51 AM Simon Taeter @.***> wrote:

So I tried following the overview https://google.github.io/dopamine/docs/ and - to be honest - this has been a pain.

The entry point to the standard Atari 2600 experiment is dopamine/discrete_domains/train.py. To run the basic DQN agent,

python -um dopamine.discrete_domains.train \ --base_dir /tmp/dopamine_runs \ --gin_files dopamine/agents/dqn/configs/dqn.gin

This raises loads of errors; first all the missing imports (I fixed those using pipreqs) then Gym complaining it doesn't distribute ROMs anymore:

gym.error.Error: We're Unable to find the game "Pong". Note: Gym no longer distributes ROMs. If you own a license to use the necessary ROMs for research purposes you can download them via pip install gym[accept-rom-license]. Otherwise, you should try importing "Pong" via the command ale-import-roms. If you believe this is a mistake perhaps your copy of "Pong" is unsupported. To check if this is the case try providing the environment variable PYTHONWARNINGS=default::ImportWarning:ale_py.roms. For more information see: https://github.com/mgbellemare/Arcade-Learning-Environment#rom-management

I realized all of this should run onto the core and atari containers that I created but I don't see where this is explained.

So I started looking at the colabs hoping to find some more infos but I got loads of errors...

agents.ipynb https://colab.research.google.com/github/google/dopamine/blob/master/dopamine/colab/agents.ipynb in Load baseline data

TypeError Traceback (most recent call last)

in () 1 # @title Load baseline data 2 get_ipython().system('gsutil -q -m cp -R gs://download-dopamine-rl/preprocessed-benchmarks/* /content/') ----> 3 experimental_data = colab_utils.load_baselines('/content')

1 frames

/usr/lib/python3.7/copyreg.py in _reconstructor(cls, base, state) 41 def _reconstructor(cls, base, state): 42 if base is object: ---> 43 obj = object.new(cls) 44 else: 45 obj = base.new(cls, state)

TypeError: object.new(BlockManager) is not safe, use BlockManager.new()

load_statistics.ipynb https://colab.research.google.com/github/google/dopamine/blob/master/dopamine/colab/load_statistics.ipynb in Load the baseline data

TypeError Traceback (most recent call last)
in () 2 3 get_ipython().system('gsutil -q -m cp -R gs://download-dopamine-rl/preprocessed-benchmarks/* /content/') ----> 4 experimental_data = colab_utils.load_baselines('/content') 1 frames /usr/lib/python3.7/copyreg.py in _reconstructor(cls, base, state) 41 def _reconstructor(cls, base, state): 42 if base is object: ---> 43 obj = object.__new__(cls) 44 else: 45 obj = base.__new__(cls, state) TypeError: object.__new__(BlockManager) is not safe, use BlockManager.__new__() agent_visualizer.ipynb in Generate the video pygame 2.1.2 (SDL 2.0.16, Python 3.7.13) Hello from the pygame community. https://www.pygame.org/contribute.html WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/tensorflow/python/compat/v2_compat.py:107: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version. Instructions for updating: non-resource variables are not supported in the long term --------------------------------------------------------------------------- Exception Traceback (most recent call last) [](https://localhost:8080/#) in () 4 example_viz_lib.run(agent='rainbow', game='SpaceInvaders', num_steps=num_steps, 5 root_dir='/tmp/agent_viz', restore_ckpt='/tmp/tf_ckpt-199', ----> 6 use_legacy_checkpoint=True) 15 frames [/usr/local/lib/python3.7/dist-packages/atari_py/games.py](https://localhost:8080/#) in get_game_path(game_name) 18 path = os.path.join(_games_dir, game_name) + ".bin" 19 if not os.path.exists(path): ---> 20 raise Exception('ROM is missing for %s, see https://github.com/openai/atari-py#roms for instructions' % (game_name,)) 21 return path 22 Exception: ROM is missing for space_invaders, see https://github.com/openai/atari-py#roms for instructions In call to configurable 'create_atari_environment' () In call to configurable 'Runner' () agent_visualizer.ipynb in Generate video WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/tensorflow/python/compat/v2_compat.py:107: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version. Instructions for updating: non-resource variables are not supported in the long term --------------------------------------------------------------------------- Exception Traceback (most recent call last) [](https://localhost:8080/#) in () 12 gin.parse_config(config) 13 runner = example_viz_lib.MyRunner(base_dir, '/tmp/ckpt.199', ---> 14 AGENT_CREATORS[agent]) 15 runner.visualize(base_dir / 'images', num_global_steps=200) 13 frames [/usr/local/lib/python3.7/dist-packages/atari_py/games.py](https://localhost:8080/#) in get_game_path(game_name) 18 path = os.path.join(_games_dir, game_name) + ".bin" 19 if not os.path.exists(path): ---> 20 raise Exception('ROM is missing for %s, see https://github.com/openai/atari-py#roms for instructions' % (game_name,)) 21 return path 22 Exception: ROM is missing for breakout, see https://github.com/openai/atari-py#roms for instructions In call to configurable 'create_atari_environment' () In call to configurable 'Runner' () happily tensorboard.ipynb functions properly and cartpole.ipynb too. Overall, all of this makes for a very poor introduction to dopamine. — Reply to this email directly, view it on GitHub , or unsubscribe . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

hi simon, the metrics issue should be resolved now, and for the ROMs you should be able to get that working with pip install gym[accept-rom-license] (this is an issue that is separate from us).

i'll look into the colab issues next week, but hopefully this unblocks you for now!

On Thu, May 19, 2022 at 9:51 AM Simon Taeter @.***> wrote:

So I tried following the overview https://google.github.io/dopamine/docs/ and - to be honest - this has been a pain.

The entry point to the standard Atari 2600 experiment is dopamine/discrete_domains/train.py. To run the basic DQN agent,

python -um dopamine.discrete_domains.train \ --base_dir /tmp/dopamine_runs \ --gin_files dopamine/agents/dqn/configs/dqn.gin

This raises loads of errors; first all the missing imports (I fixed those using pipreqs) then Gym complaining it doesn't distribute ROMs anymore:

gym.error.Error: We're Unable to find the game "Pong". Note: Gym no longer distributes ROMs. If you own a license to use the necessary ROMs for research purposes you can download them via pip install gym[accept-rom-license]. Otherwise, you should try importing "Pong" via the command ale-import-roms. If you believe this is a mistake perhaps your copy of "Pong" is unsupported. To check if this is the case try providing the environment variable PYTHONWARNINGS=default::ImportWarning:ale_py.roms. For more information see: https://github.com/mgbellemare/Arcade-Learning-Environment#rom-management

I realized all of this should run onto the core and atari containers that I created but I don't see where this is explained.

So I started looking at the colabs hoping to find some more infos but I got loads of errors...

agents.ipynb https://colab.research.google.com/github/google/dopamine/blob/master/dopamine/colab/agents.ipynb in Load baseline data

TypeError Traceback (most recent call last)

in () 1 # @title Load baseline data 2 get_ipython().system('gsutil -q -m cp -R gs://download-dopamine-rl/preprocessed-benchmarks/* /content/') ----> 3 experimental_data = colab_utils.load_baselines('/content')

1 frames

/usr/lib/python3.7/copyreg.py in _reconstructor(cls, base, state) 41 def _reconstructor(cls, base, state): 42 if base is object: ---> 43 obj = object.new(cls) 44 else: 45 obj = base.new(cls, state)

TypeError: object.new(BlockManager) is not safe, use BlockManager.new()

load_statistics.ipynb https://colab.research.google.com/github/google/dopamine/blob/master/dopamine/colab/load_statistics.ipynb in Load the baseline data

TypeError Traceback (most recent call last)
in () 2 3 get_ipython().system('gsutil -q -m cp -R gs://download-dopamine-rl/preprocessed-benchmarks/* /content/') ----> 4 experimental_data = colab_utils.load_baselines('/content') 1 frames /usr/lib/python3.7/copyreg.py in _reconstructor(cls, base, state) 41 def _reconstructor(cls, base, state): 42 if base is object: ---> 43 obj = object.__new__(cls) 44 else: 45 obj = base.__new__(cls, state) TypeError: object.__new__(BlockManager) is not safe, use BlockManager.__new__() agent_visualizer.ipynb in Generate the video pygame 2.1.2 (SDL 2.0.16, Python 3.7.13) Hello from the pygame community. https://www.pygame.org/contribute.html WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/tensorflow/python/compat/v2_compat.py:107: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version. Instructions for updating: non-resource variables are not supported in the long term --------------------------------------------------------------------------- Exception Traceback (most recent call last) [](https://localhost:8080/#) in () 4 example_viz_lib.run(agent='rainbow', game='SpaceInvaders', num_steps=num_steps, 5 root_dir='/tmp/agent_viz', restore_ckpt='/tmp/tf_ckpt-199', ----> 6 use_legacy_checkpoint=True) 15 frames [/usr/local/lib/python3.7/dist-packages/atari_py/games.py](https://localhost:8080/#) in get_game_path(game_name) 18 path = os.path.join(_games_dir, game_name) + ".bin" 19 if not os.path.exists(path): ---> 20 raise Exception('ROM is missing for %s, see https://github.com/openai/atari-py#roms for instructions' % (game_name,)) 21 return path 22 Exception: ROM is missing for space_invaders, see https://github.com/openai/atari-py#roms for instructions In call to configurable 'create_atari_environment' () In call to configurable 'Runner' () agent_visualizer.ipynb in Generate video WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/tensorflow/python/compat/v2_compat.py:107: disable_resource_variables (from tensorflow.python.ops.variable_scope) is deprecated and will be removed in a future version. Instructions for updating: non-resource variables are not supported in the long term --------------------------------------------------------------------------- Exception Traceback (most recent call last) [](https://localhost:8080/#) in () 12 gin.parse_config(config) 13 runner = example_viz_lib.MyRunner(base_dir, '/tmp/ckpt.199', ---> 14 AGENT_CREATORS[agent]) 15 runner.visualize(base_dir / 'images', num_global_steps=200) 13 frames [/usr/local/lib/python3.7/dist-packages/atari_py/games.py](https://localhost:8080/#) in get_game_path(game_name) 18 path = os.path.join(_games_dir, game_name) + ".bin" 19 if not os.path.exists(path): ---> 20 raise Exception('ROM is missing for %s, see https://github.com/openai/atari-py#roms for instructions' % (game_name,)) 21 return path 22 Exception: ROM is missing for breakout, see https://github.com/openai/atari-py#roms for instructions In call to configurable 'create_atari_environment' () In call to configurable 'Runner' () happily tensorboard.ipynb functions properly and cartpole.ipynb too. Overall, all of this makes for a very poor introduction to dopamine. — Reply to this email directly, view it on GitHub , or unsubscribe . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

google / dopamine

Poor introduction to dopamine #197