search

Bam4d / Griddly

A grid-world game engine for game AI research
https://griddly.readthedocs.io
MIT License
235 stars 24 forks source link

Feature Request: upgrade from gym to gymnasium #279

Open elliottower opened 1 year ago

elliottower commented 1 year ago

Hi, sorry if feature requests are not accepted (close this if so), but I was wondering if it would be possible to upgrade this repo from gym to gymnasium?

Gymnasium a maintained fork of openai gym and is designed as a drop-in replacement (import gym -> import gymnasium as gym). Beyond just bugfixes, many RL training libraries have also switched (rllib, tianshou, CleanRL), or are planning to switch ((stable-baselines3](https://github.com/DLR-RM/stable-baselines3/blob/e5deeed16efb57c34ccdcb14692439154d970527/docs/guide/install.rst#bleeding-edge-version)). It would be great if users could train agents on Griddly using the latest models and features from these libraries (e.g., scalable distributed training/model serving using Ray/RLlib).

For information about upgrading and compatibility, see migration guide and gym compatibility. The main difference is the API has switched to returning truncated and terminated, rather than done, in order to give more information and mitigate edge case issues (for example, many popular tutorials/implementations of Q learning using gym were actually incorrect because of done, there will be an upcoming blog post explaining more details about this on the Farama site (https://farama.org/blog).

We would love to add Griddly to our list of third party environments, but are only including repositories which are compatible with gymnasium.

We have discussed making a Shimmy wrapper and that is a possibility, but it would be best if it were done here so that more people could see it. Gymnasium doesn't directly support multi-agent environments, so it would be most ideal to make a wrapper to convert that setting into a PettingZoo environment instead. I'm personally not familiar with the Griddly codebase so it's hard for me to tell how difficult it would be to upgrade, just making this to inquire about if there are any plans or if you would be open to accepting PRs or helping with any questions we have if we do end up making a shimmy wrapper, for example.

Bam4d commented 1 year ago

Hey @elliottower yes I'm keen to upgrade to gymnasium. there's also many more upgrades to do on the python side that I'm planning like using mypy and poetry.

As for pettingzoo wrappers, I'm keen to do that also, but just need to set aside some time or find some people to help out :)

elliottower commented 1 year ago

Hey @elliottower yes I'm keen to upgrade to gymnasium. there's also many more upgrades to do on the python side that I'm planning like using mypy and poetry.

As for pettingzoo wrappers, I'm keen to do that also, but just need to set aside some time or find some people to help out :)

Good to hear, if you have any questions in the future feel free to reply here or join the farama discord and reach out there. As a personal note poetry is awesome and static typing with mypy is very nice as well. Adding those and pre commit hooks made my personal projects go so much smoother. Edit: actually my mistake, I use pyright which is a bit more full featured than mypy: https://github.com/microsoft/pyright/blob/main/docs/mypy-comparison.md

catanzaromj commented 1 year ago

Hey @Bam4d @elliottower! I am also very much interested in using griddly with PettingZoo for multiagent RL. I think this would also require increasing the ray[rllib]==2.1.0 dependency. Has there been any development towards this already? Is there an active branch where this is being investigated? I am happy to help out in terms of development, especially on the python side.

Bam4d commented 1 year ago

Hey @catanzaromj there's currently a branch that is nearly finished that's upgrading everything to gymnasium and this bumps rllib to 2.4.0 also. (also bumping a whole bunch of other things using poetry and pyprojecct.toml for example)

We'd love to have a PettingZoo wrapper for Griddly, If you're interested in helping you can branch from the gymnasium branch #282.

In terms of pettingzoo integration, I'd have a look at how griddly and rllib already integrates for multi-agent environments. The order of execution of agents is the same as the order that they are presnted in the step() function array. Griddly also has support for "parallel" execution of agents in which it will randomize the execution order.

elliottower commented 1 year ago

Make sure to use gymnasium v26 as that’s what RLlib currently uses. They are in the process of updating but currently it’s not possible to use together. On Thu, Jun 1, 2023 at 10:48 AM Chris Bamford @.***> wrote:

Hey @catanzaromj https://github.com/catanzaromj there's currently a branch that is nearly finished that's upgrading everything to gymnasium and this bumps rllib to 2.4.0 also. (also bumping a whole bunch of other things using poetry and pyprojecct.toml for example)

We'd love to have a PettingZoo wrapper for Griddly, If you're interested in helping you can branch from the gymnasium branch #282 https://github.com/Bam4d/Griddly/pull/282.

In terms of pettingzoo integration, I'd have a look at how griddly and rllib already integrates for multi-agent environments. The order of execution of agents is the same as the order that they are presnted in the step() function array. Griddly also has support for "parallel" execution of agents in which it will randomize the execution order.

— Reply to this email directly, view it on GitHub https://github.com/Bam4d/Griddly/issues/279#issuecomment-1572198863, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHVPVA66WQE7BP7SN4BKJN3XJCTSRANCNFSM6AAAAAAWFXKBH4 . You are receiving this because you were mentioned.Message ID: @.***>