[RLlib] Replace Custom Multiagent API with PettingZoo's

ray-project / ray

Ray is an AI compute engine. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

https://ray.io

Apache License 2.0

33.56k stars 5.7k forks source link

[RLlib] Replace Custom Multiagent API with PettingZoo's #23975

Open jkterry1 opened 2 years ago

jkterry1 commented 2 years ago

Description

Right now, PettingZoo serves as something akin to a multi-agent version of Gym, with support for around a multi-agent dozen learning libraries and 25+ custom environments, which makes it the most used multi-agent RL API, and the only really standard one. PettingZoo's also fairly mature, and by far the largest most complex use of multi-agent RL in RLlib (Neural MMO) has been using it for quite awhile.

Switching RLlib to use the PettingZoo API should be fairly easy, increase compatibility for different environments, improve standardization in RL, and save development costs and time for the RLlib maintainers. I spoke to @richardliaw and a few others about this on a call a few months ago, and after internal review everything looked doable, but I haven't heard anything about this since, so I just wanted to follow up here.

cc @sven1977 @avnishn @jsuarez5341 @pseudo-rnd-thoughts

Use case

No response

avnishn commented 2 years ago

Hi @jkterry1,

We have carefully studied this as a team and will respond offline.

Thanks, @avnishn

christy commented 2 years ago

From RLlib office hours May 24th, youtube video - I heard these next steps, please correct/add:

RLlib team: Consider/discuss how such a change in API could fit with ML models that have to be supported too, as well as RL models. Setup time to follow up the discussion with PZ contributors.
PettingZoo team: 44:41 Give RLlib team written list of API planned changes and approx dates. Put list here in this GH Issue, please?

stale[bot] commented 2 years ago

Hi, I'm a bot from the Ray team :)

To help human contributors to focus on more relevant issues, I will automatically add the stale label to issues that have had no activity for more than 4 months.

If there is no further activity in the 14 days, the issue will be closed!

If you'd like to keep the issue open, just leave any comment, and the stale label will be removed!
If you'd like to get more attention to the issue, please tag one of Ray's contributors.

You can always ask for help on our discussion forum or Ray's public slack channel.

jkterry1 commented 2 years ago

Hey, FYI we recently cut a release that made the last of the planned core API changes.

gresavage commented 7 months ago

Hey, I really like this idea! I hope this can be rolled into the core API soon :)

I am willing to work on a PR for this if you would like... but my guess is that it would probably be best to wait until the new API stack is finished.