search

sail-sg / envpool

C++-based high-performance parallel environment execution engine (vectorized env) for general RL environments.
https://envpool.readthedocs.io
Apache License 2.0
1.09k stars 100 forks source link

[Feature Request] support for Python environments such as minigrid #227

Closed le-horizon closed 1 year ago

le-horizon commented 1 year ago

Motivation

Currently python environments are first converted to c++ code before being supported. This isn't very scalable given all the environments which need to be translated, plus writing environment code in python has its advantages such as readability and easier to modify.

We are wondering if it is possible to directly support running python code. It's a bit tricky because the current architecture seems to rely on a threadpool. There may need to be support for a process pool before this can happen?

Solution

Support python environments directly.

le-horizon commented 1 year ago

It's great that envpool now has minigrid envs. However, the feature request is to support python envs in general, not reimplementing the python envs in c++. Does that make sense? Thanks.

mavenlin commented 1 year ago

It's great that envpool now has minigrid envs. However, the feature request is to support python envs in general, not reimplementing the python envs in c++. Does that make sense? Thanks.

Hi @le-horizon, it is doable, but probably it will have no acceleration compared to python vectorized envs. As envpool uses multithreading which python is not capable of.

The situation may change when GIL is removed in the coming python releases. However, even in that case, it would be better using python directly. Adding it under envpool would mean adding a python -> c++ -> python indirection.

With the above being said, it may still be favorable to wrap the existing VecEnv interface under envpool api. We would be happy to accept any contribution working towards that.

le-horizon commented 1 year ago

I see, thanks for the explanation, @mavenlin ! I guess if the time savings are mostly from reimplementing the environments from python to C++, instead of the threading and I/O buffering to and from the environments, then it doesn't make sense to do it on process wrapped python environments. If in the future, someone comes up with a good python compiler, then I guess we don't need to translate all the code into C++ any more.