search

Replicable-MARL / MARLlib

One repository is all that is necessary for Multi-agent Reinforcement Learning (MARL)
https://marllib.readthedocs.io
MIT License
917 stars 146 forks source link

[Proposal] Update PettingZoo version and use Gymnasium #182

Open elliottower opened 1 year ago

elliottower commented 1 year ago

Hi all, I noticed this library uses a very old version of pettingzoo which uses the unmaintained gym library, I can help with any upgrade issues but you should really update both because we have made 12 major updates since v12 and the most recent versions of environments fix many bugs. RLlib, Tianshou, CleanRL and SB3 already are compatible and we have tutorials on our site. We would love to feature your library and add a tutorial as well if you are able to update.

The main thing that will need to change is done is now two signals: termination and truncation. See https://gymnasium.farama.org/content/migration-guide/

Theohhhu commented 1 year ago

I've incorporated an example based on the version 1.23.1 of PettingZoo in pull request #184. You can access the related documentation here.

Additionally, I've made the decision to temporarily lock the Ray/RLlib version to 1.8 due to time constraints, as updating to the latest version may require significant effort. To ensure compatibility between the latest PettingZoo and the older Ray/RLlib, I've undertaken some manual adjustments, such as transforming Gymnasium's space to the legacy Gym's space, which can be found in this code line.

I would greatly appreciate any feedback you may have on this work and welcome contributions in any style to enhance it further.

elliottower commented 1 year ago

Great to hear, definitely a step in the right direction. Totally understandable with regards to extra effort/troubleshooting time.

For what it's worth RLlib added official Gymnasium support and the current nightly release ray 3.0.0 dev (https://docs.ray.io/en/latest/ray-overview/installation.html#daily-releases-nightlies) works for our RLlib tutorials, and I know their devs did extensive testing to ensure that it works for Gymnasium's different envs. The next major release should have things ironed out and be more reliable. Maybe a new contributor could help out with testing these things further.

elliottower commented 11 months ago

Hey @Theohhhu, a user on our discord asked if it is possible to use MARLlib with custom pettingzoo environments? I see in the links above there’s a few examples like SISL as well as some gymnasium robotics environments, seems like it should be possible but not sure how to help them.

Was thinking if it is possible, that would be a good tutorial for new users to use your library, if you wanted to make one we could add it on to PettingZoo’s site as well (without this though id surely get more questions about how to adapt their custom env, and be unable to help).

As a side note as well 1.23 was the only breaking changes we have planned for a while, so you could loosen the requirement to be greater than or equal to 1.23.1 if you wanted to (1.24 is out but is fully backwards compatible besides Python 3.11 support and dropping 3.7 as it has reached end of life)