How to apply RandomUniformScaleReward wrapper to vectorized environments?

[Karlheinzniebuhr]

I have a vectorized environment with multiple instances of CryptoEnv. I want to use the RandomUniformScaleReward wrapper to scale the rewards by a random factor sampled from a uniform distribution. I am not sure what is the best way to apply the wrapper to the vectorized environment. Should I wrap the vectorized environment as a whole, or should I wrap each individual environment inside the lambda function?

n_envs = 256

# Create a list of environment constructors
envs = [lambda: CryptoEnv(df=gymdf, window_size=window_size, frame_bound=training_frame_bound) for _ in range(n_envs)]

# Create the vectorized environment
envs = DummyVecEnv(envs)

# Option 1: Wrap the vectorized environment with RandomUniformScaleReward
envs = RandomUniformScaleReward(env=envs, noise_rate=0.01, low=0.9, high=1.1)

# Option 2: Wrap each individual environment with RandomUniformScaleReward inside the lambda function
envs = [lambda: RandomUniformScaleReward(env=CryptoEnv(df=gymdf, window_size=window_size, frame_bound=training_frame_bound), noise_rate=0.01, low=0.9, high=1.1) for _ in range(n_envs)]

envs = DummyVecEnv(envs)

[raadk]

Something like the below should work, though you may want to set your seeds appropriately.

import gymnasium as gym # 0.28.1
from noisyenv.wrappers import RandomUniformScaleReward

def make_env():
    # Inspired from https://github.com/vwxyzjn/cleanrl/blob/master/cleanrl/ppo.py
    def thunk():
        base_env = gym.make("MountainCar-v0") 
        env = RandomUniformScaleReward(env=base_env, noise_rate=1.0, low=0.9, high=1.1)
        return env
    return thunk

n_envs = 256
envs = gym.vector.SyncVectorEnv([make_env() for i in range(n_envs)])