search

ray-project / ray

Ray is a unified framework for scaling AI and Python applications. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
https://ray.io
Apache License 2.0
31.97k stars 5.45k forks source link

[RLlib] New API Stack does not support RLlib's own VectorEnv #45802

Open TooSchoolForCool opened 3 weeks ago

TooSchoolForCool commented 3 weeks ago

What happened + What you expected to happen

I implement a vectorized environment drived from ray.rllib.env.vector_env.VectorEnv.

However, we I use the new API stack, it says

AttributeError: 'MecKinovaVecEnv' object has no attribute 'unwrapped'

The vectorized environment work as expected when I use the old API stack.

It seems new API stack does not support RLlib's own VectorEnv, will they support it in future?

Versions / Dependencies

Reproduction script

There is no bug when I use following configuration:

# config 1 old API stack: No bug
config = (
    get_trainable_cls(args.run)
    .get_default_config()
    .environment(MecKinovaVecEnv, env_config={"env_config": None, "num_envs": 64})
    .framework(args.framework)
    .rollouts(num_rollout_workers=2, num_envs_per_worker=64)
    .resources(num_gpus=0.5, num_gpus_per_worker=0.25)
    .training(lr=5e-5)
)

# config 2 new API stack without enabling EnvRunners: No bug
config = (
    get_trainable_cls(args.run)
    .get_default_config()
    .api_stack(enable_rl_module_and_learner=True)
    .environment(MecKinovaVecEnv, env_config={"env_config": None, "num_envs": 64})
    .framework(args.framework)
    .env_runners(num_env_runners=2, num_envs_per_env_runner=64, num_gpus_per_env_runner=0.25)
    .learners(num_learners=0, num_gpus_per_learner=0.5)
    .training(lr=5e-5)
)

However, we I enable the EnvRunners, it raise ERROR says object has no attribute 'unwrapped', the configuration is as follows:

# config 3 new API stack enabling EnvRunners: Raise `object has no attribute 'unwrapped`
config = (
    get_trainable_cls(args.run)
    .get_default_config()
    .api_stack(enable_rl_module_and_learner=True, enable_env_runner_and_connector_v2=True)
    .environment(MecKinovaVecEnv, env_config={"env_config": None, "num_envs": 64})
    .framework(args.framework)
    .env_runners(num_env_runners=2, num_envs_per_env_runner=64, num_gpus_per_env_runner=0.25)
    .learners(num_learners=0, num_gpus_per_learner=0.5)
    .training(lr=5e-5, model={"uses_new_env_runners": True})
)

Issue Severity

Medium: It is a significant difficulty but I can work around it.

ciroaceto commented 3 weeks ago

If you check RLlib docs on the comparison of the old and new stack here, you'll see it uses gym.vector.Env to vectorize environments. So the answer to the future support for VectorEnv is likely no.