search

google-deepmind / meltingpot

A suite of test scenarios for multi-agent reinforcement learning.
Apache License 2.0
614 stars 122 forks source link

Issue 25: Extend rllib examples to save, load, and render bots #31

Closed willis-richard closed 2 years ago

willis-richard commented 2 years ago

This patch uses ray.tune to save the trained models from self_play_train.py and allows you to view their behaviour in a pygame instance by running view_model.py.

Disclaimer: I am only ~1 month familiar with RLLib and Melting Pot.

Note: It is unfortunate that I needed to pass RayModelPolicy a full rllib.agents.trainer.Trainer instance, rather than just a rllib.policy.policy.Policy. This is because rllib.agents.trainer does some pre-processing of the inputs before forwarding to the policy (see lines 1452-1457 in Ray 1.11.0).

Also, although against the spirit of the library, I wanted to demonstrate that the RayModelPolicy could play against the PermissiveModelPolicy. I was not able to do this however, as replacing some of the bots in view_model.py with bots created via: bot_factory.build(bot_factory.get_config("ah3gs_bot_finding_berry_two_the_most_tasty_0")) Resulted in errors about the observations that they received. Therefore rllib bots and existing meltingpots cannot be mixed at the moment.

Finally, would you be open to answering some questions I have about use of this repo? I intend to make some modifications to meltingpot for my experiments and I would appreciate it if I could run my proposed changes by you.

duenez commented 2 years ago

Regarding using our saved model bots for evaluation, it would work if you instead just instantiate an scenario rather than the substrate. Then the appropriate wrappers would be put around the bots, and you'd be able to get an environment that you only connect your RLLib policies to. Then you can just render the WORLD.RGB which would show all players. Makes sense?

willis-richard commented 2 years ago

Thank you for the pointer. I think it makes sense, though I will leave this as future work for now. (First I need successfully train some bots!)

jzleibo commented 2 years ago

Hi Richard!

On Tue, May 3, 2022 at 11:38 AM Richard Willis @.***> wrote:

This patch uses ray.tune to save the trained models from self_play_train.py and allows you to view their behaviour in a pygame instance by running view_model.py.

Disclaimer: I am only ~1 month familiar with RLLib and Melting Pot.

Note: It is unfortunate that I needed to pass RayModelPolicy a full rllib.agents.trainer.Trainer instance, rather than just a rllib.policy.policy.Policy. This is because rllib.agents.trainer does some pre-processing of the inputs before forwarding to the policy (see lines 1452-1457 in Ray 1.11.0).

Also, although against the spirit of the library, I wanted to demonstrate that the RayModelPolicy could play against the PermissiveModelPolicy. I was not able to do this however, as replacing some of the bots in view_model.py with bots created via:

bot_factory.build(bot_factory.get_config("ah3gs_bot_finding_berry_two_the_most_tasty_0")) Resulted in errors about the observations that they received. Therefore rllib bots and existing meltingpots cannot be mixed at the moment.

Finally, would you be open to answering some questions I have about use of this repo? I intend to make some modifications to meltingpot for my experiments and I would appreciate it if I could run my proposed changes by you.

I somehow lost track of this thread -- I remembered that someone had said they were planning to make some modifications to meltingpot for their experiments and wanted to run the changes by us. And I was even actively searching around for this message thread to reply before, but I couldn't for the life of me find it then.! Concluded I must have hallucinated the original message.. Anyway, I've found it now :). Let me know if you still want to chat sometime. I'm happy to schedule a call.

On Tue, May 3, 2022 at 11:38 AM Richard Willis @.***> wrote:

This patch uses ray.tune to save the trained models from self_play_train.py and allows you to view their behaviour in a pygame instance by running view_model.py.

Disclaimer: I am only ~1 month familiar with RLLib and Melting Pot.

Note: It is unfortunate that I needed to pass RayModelPolicy a full rllib.agents.trainer.Trainer instance, rather than just a rllib.policy.policy.Policy. This is because rllib.agents.trainer does some pre-processing of the inputs before forwarding to the policy (see lines 1452-1457 in Ray 1.11.0).

Also, although against the spirit of the library, I wanted to demonstrate that the RayModelPolicy could play against the PermissiveModelPolicy. I was not able to do this however, as replacing some of the bots in view_model.py with bots created via:

bot_factory.build(bot_factory.get_config("ah3gs_bot_finding_berry_two_the_most_tasty_0")) Resulted in errors about the observations that they received. Therefore rllib bots and existing meltingpots cannot be mixed at the moment.

Finally, would you be open to answering some questions I have about use of this repo? I intend to make some modifications to meltingpot for my experiments and I would appreciate it if I could run my proposed changes by you.

You can view, comment on, or merge this pull request online at:

https://github.com/deepmind/meltingpot/pull/31 Commit Summary

File Changes

(4 files https://github.com/deepmind/meltingpot/pull/31/files)

Patch Links:

— Reply to this email directly, view it on GitHub https://github.com/deepmind/meltingpot/pull/31, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAHFJWS6VX4CHRHQCF77JNLVID64BANCNFSM5U6QPFYA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

willis-richard commented 2 years ago

@jzleibo thank you, that would be great. I shall send an email to the address given in your personal blog.