Feature request: upgrade from gym to gymnasium

[elliottower]

Hi, I was wondering if it would be possible to update MineRL to use gymnasium rather than gym? We would like to list this challenge in this list of third party environments, but are only adding environments which use gymnasium. I am also personally very interested in testing training agents in Minecraft, and would like to use newer models from training libraries like RLlib.

Gymnasium a maintained fork of openai gym and is designed as a drop-in replacement (import gym -> import gymnasium as gym). Beyond just bugfixes, many RL training libraries have also switched (rllib, tianshou, CleanRL), or are planning to switch (stable-baselines3). It would be great if users could train agents on Habitat using the latest models and features from these libraries (e.g., scalable distributed training/model serving using Ray/RLlib).

For information about upgrading and compatibility, see migration guide and gym compatibility. The main difference is the API has switched to returning truncated and terminated, rather than done, in order to give more information and mitigate edge case issues (for example, many popular tutorials/implementations of Q learning using gym were actually incorrect because of done, there will be an upcoming blog post explaining more details about this on the Farama site (https://farama.org/blog).

[Miffyli]

Hey! We unfortunately from MineRL side do not have bandwidth to update code in such major ways as maintainer team is now working elsewhere. Updates would be cool for MineRL, but the different versions of MineRL used Gym in rather nasty ways which might not be migrateable to gymnasium 😅. Ideally, all major versions should be supported (people might want to run things on them), and meanwhile, there are bigger things to fix if anybody finds the time (e.g., more sane installation process).

[elliottower]

Hey! We unfortunately from MineRL side do not have bandwidth to update code in such major ways as maintainer team is now working elsewhere. Updates would be cool for MineRL, but the different versions of MineRL used Gym in rather nasty ways which might not be migrateable to gymnasium 😅. Ideally, all major versions should be supported (people might want to run things on them), and meanwhile, there are bigger things to fix if anybody finds the time (e.g., more sane installation process).

Thanks for getting back to me. I don’t currently have the bandwith to help out but i and a few friends are interested in working with MineRL in the future so could potentially help with that. Definitely makes sense to support both for backwards compatibility but would be a bit of work. VizDoom for example handles both gym and gymnasium I know.

Anyways maybe we can leave this open in case any new contributors want to help out? I can definitely give guidance about the details of adapting code to work with gymnasium if need be.

[Bpoole908]

@elliottower @Miffyli I might be potentially interested in converting the code Gymnasium. I'm still relatively new to the code base but have been trying to convert some other repos to Gymnasium as well.

Is there any estimate on what exactly needs to be done or any specific areas that are likely to be a problem? Additionally, if adding backwards compatibility proves too difficult, maybe a quick fix for now could be keeping Gymnasium changes in a separate branch and adding backwards compatibility in later?

[Miffyli]

@Bpoole908 You are welcome to give it a try, but I can not promise a hasty PR review on this :). Backwards compatibility is a must, since there are probably tons of projects out there which did not specify specific MineRL versions... Another major issue is that there is at least two major MineRL versions we would like to keep alive: v0.4.4 and v1.0>. They have quite a bit different codes, although Python side is mostly same.

Anywho, the core issue comes from the way Gym environments are "composed" together piece by piece from different things. I can not remember specifics, but you can get the error by installing gym>0.19 and trying to create the env. Tbh I would be happier with PRs that fix the code for newest version of gym first, and then see about Gymnasium update.

[elliottower]

For what it's worth, if you can update the code to work with gym v26 then the transition to gymnasium will be very simple. Sounds a bit hectic having to support two different versions but I'm sure there's a way to do it. Would be happy to take a look at a PR with the gymnasium-related updates (mainly it's just changing from done to termination, truncation as I said above, and env.reset() always returning obs and info as a tuple rather than taking the return_info argument)