Open vwxyzjn opened 1 year ago
Ah very cool @vwxyzjn! I'm learning a bit on how this is different from MultiProcessRLEnv
(source). In general I agree yes that this would be good to support, all the best RL infrastructure's I know have been going async like this.
I think @edbeeching will comment when he's back in a couple days.
Hello, this work looks pretty cool and looking forward to using it in the future.
I was wondering if you would be interested in implementing EnvPool's Asynchronous API, which looks like below:
The general idea is to return a subset of the environments for the agent to sample actions while the environments execute other actions. This approach should scale considerably better, primarily when the engine backend uses
socket
(#219). In CleanRL we have a fast PPO implementation prototype that leverages this async API (see code here)https://github.com/Farama-Foundation/Gymnasium/pull/98 also contains an example of implementing this type of Async API with existing vectorized environments.