Open Mayankm96 opened 1 year ago
A third option to consider:
Simply define it as a Protocol
. Something like EnvCompatible
.
Over the years, I have run into countless issues I can trace back to the "kitchen sink" structure of gym
. Don't get me wrong, I am extremely grateful for the step towards standardization that it catalyzed, but it's clear that when it was originally formulated, it was meant to be a standalone repository. I'm always looking to shy away from taking on unneeded dependencies since this can do a lot of harm to the lifecycle of the codebase.
I believe in the modern days, gym
is used more like an interface to ensure that libraries for RL (both on the algorithmic side and on the dynamics side) agree on the correct "protocol" of methods that should be made available.
The best part is using a Protocol
definition is also backward compatible as well as compatible with libraries that depend on gymnasium or whichever other flavor.
Something as simple as:
from typing import Protocol, runtime_checkable, Any, SupportsFloat, Optional
@dataclass
class StepReturn:
observation: Any
reward: SupportsFloat
terminated: bool
truncated: bool
#info: Info | None
@dataclass
class ResetReturn:
observation: Any
#info: Info | None
@runtime_checkable
class EnvCompatible(Protocol):
def step( self, action: Any ) -> StepReturn:
"""Step."""
def reset( self, seed: Optional[Any], options: Optional[Any]) -> ResetReturn:
"""Reset."""
Edit: You can also make something IsaacEnvCompatible
or RLGamesEnvCompatible
to specify additional behavior that must be satisfied. Multiple-implementation of protocols is seamless since there is no MRO to contend with (as is the case with abstract base classes).
Anyways, just something to think about.
Proposal
Gymnasium is the maintained version of OpenAI Gym that is now handled by Farama Foundation. It is currently the definition used in libraries such as rllib and tianshou, while others like stable-baselines3 will also update soon.
More recently, there has been an interesting proposal on RL environment definition in the framework
torchrl
which would fit well for our targetted applications.This thread aims to serve as a voting/staging on what would be the best environment definition to use.
Motivation
Migrating to Gym 0.28
The current Gym definition (from 0.21.0) is outdated and depends on older libraries such as
importlib-meta==4.1
package. This creates conflicts with new updated packages and it would be best to switch to the new Gym definition.Related issues: OIGE #28, Sb3 #1327
Once SB3 also upgrades to Gymnasium, then it would be best to update everything to use the latest definitions.
More information on Gymnasium:
Migrating to TorchRL definition
The environment definition
EnvBase
which relies heavily on usingtensordict
.The advantages of their base class are:
tensordict
makes it easier to work with given the scaling possible with Isaac Sim and should be more efficient/flexible thandict
oftorch.Tensor
Related Issues: torchrl #883
Effect on the remaining framework
Since there are wrappers for other RL frameworks (RL-Games and RSL-RL), this probably won't cause any breaking changes on that side. We will just need to adapt them based on the chosen
IsaacEnv
definition.Checklist