Not able to load the agent and get baseline result in the rainbow paper

[zhuyifengzju]

Hi, I was trying to load a rainbow agent form the checkpoint of breakout and want to see the results from the paper, but I was not able to make it work. It seems that the checkpoint prefix does not match the agent codebase. I am not sure if I am wrong about this. Any help would be appreciated. Thanks!

[psc-g]

hi, this is probably because our networks have moved to keras. you should be able to fix this by adding the following line to your gin file:

atari_lib.maybe_transform_variable_names.legacy_checkpoint_load = True

if you check our main page https://github.com/google/dopamine you can find more details under the What's new section. let me know if this does or does not resolve your issue.

On Mon, Dec 16, 2019 at 1:40 PM YF notifications@github.com wrote:

Hi, I was trying to load a rainbow agent form the checkpoint of breakout and want to see the results from the paper, but I was not able to make it work. It seems that the checkpoint prefix does not match the agent codebase. I am not sure if I am wrong about this. Any help would be appreciated. Thanks!

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/google/dopamine/issues/123?email_source=notifications&email_token=AE3CCMJJM7BI3RGVHAS4BNTQY7DTJA5CNFSM4J3OGG42YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IA2EWCQ, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE3CCMLQGWUF6B4J5HRUN5TQY7DTJANCNFSM4J3OGG4Q .

[zhuyifengzju]

Thanks for your reply! However, my agent is still freezing when I use the visualization. Here is what I did:

`from dopamine.utils import example_viz_lib num_steps = 1000 example_viz_lib.run(agent='rainbow', game='Seaquest', num_steps=num_steps, root_dir='./viz', restore_ckpt='./Seaquest_checkpoint/checkpoints/tf_ckpt-199')

`

And I added the legacy checkpoint load in the command line using gin-binding argument. The agent in seaquest does not move. The same thing happens to the breakout agent. If you have any thought why this is happening, please let me know. Thanks!

[psc-g]

just to clarify, when you say "does not move" does it mean it's generating many png files but in the resulting video they all appear to be the same frame?

On Mon, Dec 16, 2019 at 4:32 PM YF notifications@github.com wrote:

Thanks for your reply! However, my agent is still freezing when I use the visualization. Here is what I did:

`from dopamine.utils import example_viz_lib num_steps = 1000 example_viz_lib.run(agent='rainbow', game='Seaquest', num_steps=num_steps, root_dir='./viz', restore_ckpt='./Seaquest_checkpoint/checkpoints/tf_ckpt-199')

`

And I added the legacy checkpoint load in the command line using gin-binding argument. The agent in seaquest does not move. The same thing happens to the breakout agent. If you have any thought why this is happening, please let me know. Thanks!

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/google/dopamine/issues/123?email_source=notifications&email_token=AE3CCMJULA2NMNLC6RFRETLQY7XVXA5CNFSM4J3OGG42YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHAFMBA#issuecomment-566253060, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE3CCMPPELQGXWZ6CGKRBHLQY7XVXANCNFSM4J3OGG4Q .

[zhuyifengzju]

Yes, that is what happened

[psc-g]

can you verify that the checkpoint is being properly restored? there should be a tensorflow log printed that says something like "Successfully reloaded from ...".

On Mon, Dec 16, 2019 at 4:57 PM YF notifications@github.com wrote:

Yes, that is what happened

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/google/dopamine/issues/123?email_source=notifications&email_token=AE3CCMPVJWTDWX3ITHCNDN3QY72TPA5CNFSM4J3OGG42YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHAHWFI#issuecomment-566262549, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE3CCMJQVHV4TSAMXHTBMUTQY72TPANCNFSM4J3OGG4Q .

[zhuyifengzju]

Hi, I am checking the example_viz_lib.py file, and I found that the variables_to_restore in the reload_checkpoint function (In the MyRainbowAgent class) does not have all the variables from include_vars. I only got beta1_power:0 and beta2_power:0 variables. I feel like there is something wrong with slim.get_variables_to_restore. Could you help check if everything goes correctly on your side? Thanks!

[psc-g]

ah, i think i know what the issue is. could you replace the line

variables_to_restore = contrib_slim...

with the following line:

variables_to_restore = atari_lib.maybe_transform_variable_names(tf.all_variables(), legacy_checkpoint_load=True)

i think that should resolve your issue, but let me know if it doesn't. i will be pushing a fix to this in the next couple of days.

thanks for pointing this out!

[zhuyifengzju]

Thanks for the reply! Actually I removed ":0" and mapped new names to old names (Changing kernel to weights and bias to biases) and it worked. It seemed that ":0" is also making things failed.

[psc-g]

thanks for letting me know. i tested locally and the fix i suggested also seemed to work, so i'll be pushing that out in the next few days. do let us know if you run into any further issues!

On Wed, Dec 18, 2019 at 11:23 AM YF notifications@github.com wrote:

Thanks for the reply! Actually I removed ":0" and mapped new names to old names (Changing kernel to weights and bias to biases) and it worked. It seemed that ":0" is also making things failed.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/google/dopamine/issues/123?email_source=notifications&email_token=AE3CCMOJT5OP4P2KSBFBDBTQZJFAZA5CNFSM4J3OGG42YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHGVPQY#issuecomment-567105475, or unsubscribe https://github.com/notifications/unsubscribe-auth/AE3CCMN7OXRGSG5SDQFPE43QZJFAZANCNFSM4J3OGG4Q .

[zhuyifengzju]

Sure, thanks for answering my issue!

[psc-g]

i just pushed out a fix for this. let me know if you still run into problems by reopening this issue or raising another one.