search

google-deepmind / bsuite

bsuite is a collection of carefully-designed experiments that investigate core capabilities of a reinforcement learning (RL) agent
Apache License 2.0
1.51k stars 182 forks source link

Environment seeding #36

Closed adamxyang closed 3 years ago

adamxyang commented 3 years ago

How to set a seed in a bsuite environment instance? In the notebook, the output of sweep.SETTINGS has a seed attribute which is not None:

Loaded bsuite_id: bandit_noise/0.
bsuite_id=bandit_noise/0, settings={'noise_scale': 0.1, 'seed': 0}, num_episodes=10000
Loaded bsuite_id: bandit_noise/1.
bsuite_id=bandit_noise/1, settings={'noise_scale': 0.1, 'seed': 1}, num_episodes=10000
Loaded bsuite_id: bandit_noise/2.
bsuite_id=bandit_noise/2, settings={'noise_scale': 0.1, 'seed': 2}, num_episodes=10000
Loaded bsuite_id: bandit_noise/3.
bsuite_id=bandit_noise/3, settings={'noise_scale': 0.1, 'seed': 3}, num_episodes=10000

but when I printed it again on my own computer, seed was None (if I do that in the notebook, seed was None but there's an extra mapping_seed which was not None).

I tried two methods to seed the environment: (1) sweep.SETTINGS[bsuite_id]['seed']=0; (2) doing env.seed() after wrapping it with OpenAI env, but neither worked (multiple experiments, same seed, different results). A minimal example to demonstrate these two seeding methods are not working:

import random
import torch as t
import numpy as np
import bsuite
from bsuite import sweep
from bsuite.utils import gym_wrapper

def set_seed(seed, deterministic=True):
    random.seed(seed)
    np.random.seed(seed)
    t.manual_seed(seed)
    t.cuda.manual_seed_all(seed)
    t.cuda.manual_seed(seed)
    if deterministic:
        t.backends.cudnn.deterministic = True
        t.backends.cudnn.benchmark = False

set_seed(0)
bsuite_id = 'cartpole_swingup/0'
raw_env = bsuite.load_from_id(bsuite_id)

# method 1
sweep.SETTINGS[bsuite_id]['seed']=0
for episode in range(10):
    timestep = raw_env.reset()
    total_reward = 0
    while not timestep.last():
        action = np.random.choice(raw_env.action_spec().num_values)
        timestep = raw_env.step(action)
        total_reward += timestep.reward
    print(episode,total_reward)

# method 2
env = gym_wrapper.GymFromDMEnv(raw_env)
env.seed(seed=0)
for episode in range(10):
    timestep = env.reset()
    total_reward = 0
    done = False
    while not done:
        action = np.random.choice(raw_env.action_spec().num_values)
        sn,r,done,_ = env.step(action)
        total_reward += r
    print(episode,total_reward)
ayakayal commented 2 years ago

How to set a seed in a bsuite environment instance? In the notebook, the output of sweep.SETTINGS has a seed attribute which is not None:

Loaded bsuite_id: bandit_noise/0.
bsuite_id=bandit_noise/0, settings={'noise_scale': 0.1, 'seed': 0}, num_episodes=10000
Loaded bsuite_id: bandit_noise/1.
bsuite_id=bandit_noise/1, settings={'noise_scale': 0.1, 'seed': 1}, num_episodes=10000
Loaded bsuite_id: bandit_noise/2.
bsuite_id=bandit_noise/2, settings={'noise_scale': 0.1, 'seed': 2}, num_episodes=10000
Loaded bsuite_id: bandit_noise/3.
bsuite_id=bandit_noise/3, settings={'noise_scale': 0.1, 'seed': 3}, num_episodes=10000

but when I printed it again on my own computer, seed was None (if I do that in the notebook, seed was None but there's an extra mapping_seed which was not None).

I tried two methods to seed the environment: (1) sweep.SETTINGS[bsuite_id]['seed']=0; (2) doing env.seed() after wrapping it with OpenAI env, but neither worked (multiple experiments, same seed, different results). A minimal example to demonstrate these two seeding methods are not working:

import random
import torch as t
import numpy as np
import bsuite
from bsuite import sweep
from bsuite.utils import gym_wrapper

def set_seed(seed, deterministic=True):
    random.seed(seed)
    np.random.seed(seed)
    t.manual_seed(seed)
    t.cuda.manual_seed_all(seed)
    t.cuda.manual_seed(seed)
    if deterministic:
        t.backends.cudnn.deterministic = True
        t.backends.cudnn.benchmark = False

set_seed(0)
bsuite_id = 'cartpole_swingup/0'
raw_env = bsuite.load_from_id(bsuite_id)

# method 1
sweep.SETTINGS[bsuite_id]['seed']=0
for episode in range(10):
    timestep = raw_env.reset()
    total_reward = 0
    while not timestep.last():
        action = np.random.choice(raw_env.action_spec().num_values)
        timestep = raw_env.step(action)
        total_reward += timestep.reward
    print(episode,total_reward)

# method 2
env = gym_wrapper.GymFromDMEnv(raw_env)
env.seed(seed=0)
for episode in range(10):
    timestep = env.reset()
    total_reward = 0
    done = False
    while not done:
        action = np.random.choice(raw_env.action_spec().num_values)
        sn,r,done,_ = env.step(action)
        total_reward += r
    print(episode,total_reward)

@adamxyang Did you find solution to this as I am having the same issue. I tried fixing the seed but I am still getting different results on different runs. Thanks.