ValueError: You have passed a tuple to the predict() function instead of a Numpy array or a Dict. You are probably mixing Gym API with SB3 VecEnv API: `obs, info = env.reset()` (Gym) vs `obs = vec_env.reset()` (SB3 VecEnv).[Bug]: bug title #1950
I am running the learn.py example that came with the repo, no modifications.
It seems like the training works perfectly fine and also saves the model, but I encounter an error at that instance of predict() method. It seems that there are inconsistencies with what gym vs sb3 returns for the reset() method, hence inputting an invalid argument into predict(). I tried this bypass, and while it fixed the issue at predict(), it ran into another issue later on running evaluation.py in the stable_baselines3 package, causing me to be unable to evaluate my state. This other issue is of the same nature as it has another problem with the dimensions that are returned with reset().
I made sure that I have gymnasium support installed as well. I'm kind of at a loss for what to do here. Does anyone know if I am supposed to edit the source code in sb3?
To Reproduce
import os
import time
from datetime import datetime
import argparse
import gymnasium as gym
import numpy as np
import torch
from stable_baselines3 import PPO
from stable_baselines3.common.env_util import make_vec_env
from stable_baselines3.common.callbacks import EvalCallback, StopTrainingOnRewardThreshold
from stable_baselines3.common.evaluation import evaluate_policy
from gym_pybullet_drones.utils.Logger import Logger
from gym_pybullet_drones.envs.HoverAviary import HoverAviary
from gym_pybullet_drones.envs.MultiHoverAviary import MultiHoverAviary
from gym_pybullet_drones.utils.utils import sync, str2bool
from gym_pybullet_drones.utils.enums import ObservationType, ActionType
DEFAULT_GUI = True
DEFAULT_RECORD_VIDEO = True
DEFAULT_OUTPUT_FOLDER = 'results'
DEFAULT_COLAB = False
DEFAULT_OBS = ObservationType('kin') # 'kin' or 'rgb'
DEFAULT_ACT = ActionType('one_d_rpm') # 'rpm' or 'pid' or 'vel' or 'one_d_rpm' or 'one_d_pid'
DEFAULT_AGENTS = 2
DEFAULT_MA = False
def run(multiagent=DEFAULT_MA, output_folder=DEFAULT_OUTPUT_FOLDER, gui=DEFAULT_GUI, plot=True, colab=DEFAULT_COLAB, record_video=DEFAULT_RECORD_VIDEO, local=True):
filename = os.path.join(output_folder, 'save-'+datetime.now().strftime("%m.%d.%Y_%H.%M.%S"))
if not os.path.exists(filename):
os.makedirs(filename+'/')
if not multiagent:
train_env = make_vec_env(HoverAviary,
env_kwargs=dict(obs=DEFAULT_OBS, act=DEFAULT_ACT),
n_envs=1,
seed=0
)
eval_env = HoverAviary(obs=DEFAULT_OBS, act=DEFAULT_ACT)
else:
train_env = make_vec_env(MultiHoverAviary,
env_kwargs=dict(num_drones=DEFAULT_AGENTS, obs=DEFAULT_OBS, act=DEFAULT_ACT),
n_envs=1,
seed=0
)
eval_env = MultiHoverAviary(num_drones=DEFAULT_AGENTS, obs=DEFAULT_OBS, act=DEFAULT_ACT)
#### Check the environment's spaces ########################
print('[INFO] Action space:', train_env.action_space)
print('[INFO] Observation space:', train_env.observation_space)
#### Train the model #######################################
model = PPO('MlpPolicy',
train_env,
# tensorboard_log=filename+'/tb/',
verbose=1)
#### Target cumulative rewards (problem-dependent) ##########
if DEFAULT_ACT == ActionType.ONE_D_RPM:
target_reward = 474.15 if not multiagent else 949.5
else:
target_reward = 467. if not multiagent else 920.
callback_on_best = StopTrainingOnRewardThreshold(reward_threshold=target_reward,
verbose=1)
eval_callback = EvalCallback(eval_env,
callback_on_new_best=callback_on_best,
verbose=1,
best_model_save_path=filename+'/',
log_path=filename+'/',
eval_freq=int(1000),
deterministic=True,
render=False)
model.learn(total_timesteps=int(1e7) if local else int(1e2), # shorter training in GitHub Actions pytest
callback=eval_callback,
log_interval=100)
#### Save the model ########################################
model.save(filename+'/final_model.zip')
print(filename)
#### Print training progression ############################
with np.load(filename+'/evaluations.npz') as data:
for j in range(data['timesteps'].shape[0]):
print(str(data['timesteps'][j])+","+str(data['results'][j][0]))
############################################################
############################################################
############################################################
############################################################
############################################################
if local:
input("Press Enter to continue...")
# if os.path.isfile(filename+'/final_model.zip'):
# path = filename+'/final_model.zip'
if os.path.isfile(filename+'/best_model.zip'):
path = filename+'/best_model.zip'
else:
print("[ERROR]: no model under the specified path", filename)
model = PPO.load(path)
#### Show (and record a video of) the model's performance ##
if not multiagent:
test_env = HoverAviary(gui=gui,
obs=DEFAULT_OBS,
act=DEFAULT_ACT,
record=record_video)
test_env_nogui = HoverAviary(obs=DEFAULT_OBS, act=DEFAULT_ACT)
else:
test_env = MultiHoverAviary(gui=gui,
num_drones=DEFAULT_AGENTS,
obs=DEFAULT_OBS,
act=DEFAULT_ACT,
record=record_video)
test_env_nogui = MultiHoverAviary(num_drones=DEFAULT_AGENTS, obs=DEFAULT_OBS, act=DEFAULT_ACT)
logger = Logger(logging_freq_hz=int(test_env.CTRL_FREQ),
num_drones=DEFAULT_AGENTS if multiagent else 1,
output_folder=output_folder,
colab=colab
)
mean_reward, std_reward = evaluate_policy(model,
test_env_nogui,
n_eval_episodes=10
)
print("\n\n\nMean reward ", mean_reward, " +- ", std_reward, "\n\n")
obs = test_env.reset(seed=42, options={})
start = time.time()
for i in range((test_env.EPISODE_LEN_SEC+2)*test_env.CTRL_FREQ):
action, _states = model.predict(obs,
deterministic=True
)
obs, reward, terminated, truncated, info = test_env.step(action)
obs2 = obs.squeeze()
act2 = action.squeeze()
print("Obs:", obs, "\tAction", action, "\tReward:", reward, "\tTerminated:", terminated, "\tTruncated:", truncated)
if DEFAULT_OBS == ObservationType.KIN:
if not multiagent:
logger.log(drone=0,
timestamp=i/test_env.CTRL_FREQ,
state=np.hstack([obs2[0:3],
np.zeros(4),
obs2[3:15],
act2
]),
control=np.zeros(12)
)
else:
for d in range(DEFAULT_AGENTS):
logger.log(drone=d,
timestamp=i/test_env.CTRL_FREQ,
state=np.hstack([obs2[d][0:3],
np.zeros(4),
obs2[d][3:15],
act2[d]
]),
control=np.zeros(12)
)
test_env.render()
print(terminated)
sync(i, start, test_env.CTRL_TIMESTEP)
if terminated:
obs = test_env.reset(seed=42, options={})
test_env.close()
if plot and DEFAULT_OBS == ObservationType.KIN:
logger.plot()
if __name__ == '__main__':
#### Define and parse (optional) arguments for the script ##
parser = argparse.ArgumentParser(description='Single agent reinforcement learning example script')
parser.add_argument('--multiagent', default=DEFAULT_MA, type=str2bool, help='Whether to use example LeaderFollower instead of Hover (default: False)', metavar='')
parser.add_argument('--gui', default=DEFAULT_GUI, type=str2bool, help='Whether to use PyBullet GUI (default: True)', metavar='')
parser.add_argument('--record_video', default=DEFAULT_RECORD_VIDEO, type=str2bool, help='Whether to record a video (default: False)', metavar='')
parser.add_argument('--output_folder', default=DEFAULT_OUTPUT_FOLDER, type=str, help='Folder where to save logs (default: "results")', metavar='')
parser.add_argument('--colab', default=DEFAULT_COLAB, type=bool, help='Whether example is being run by a notebook (default: "False")', metavar='')
ARGS = parser.parse_args()
run(**vars(ARGS))
Relevant log output / Error message
Traceback (most recent call last):
File "/Users/kevinhan/gym-pybullet-drones/gym_pybullet_drones/examples/learn.py", line 199, in <module>
run(**vars(ARGS))
File "/Users/kevinhan/gym-pybullet-drones/gym_pybullet_drones/examples/learn.py", line 150, in run
action, _states = model.predict(obs,
File "/Users/kevinhan/opt/anaconda3/envs/drones/lib/python3.10/site-packages/stable_baselines3/common/base_class.py", line 556, in predict
return self.policy.predict(observation, state, episode_start, deterministic)
File "/Users/kevinhan/opt/anaconda3/envs/drones/lib/python3.10/site-packages/stable_baselines3/common/policies.py", line 357, in predict
raise ValueError(
ValueError: You have passed a tuple to the predict() function instead of a Numpy array or a Dict. You are probably mixing Gym API with SB3 VecEnv API: `obs, info = env.reset()` (Gym) vs `obs = vec_env.reset()` (SB3 VecEnv). See related issue https://github.com/DLR-RM/stable-baselines3/issues/1694 and documentation for more information: https://stable-baselines3.readthedocs.io/en/master/guide/vec_envs.html#vecenv-api-vs-gym-api
System Info
OS: macOS-10.16-x86_64-i386-64bit Darwin Kernel Version 22.5.0: Thu Jun 8 22:22:22 PDT 2023; root:xnu-8796.121.3~7/RELEASE_X86_64
Python: 3.10.14
Stable-Baselines3: 2.4.0a1
PyTorch: 2.2.2
GPU Enabled: False
Numpy: 1.26.4
Cloudpickle: 3.0.0
Gymnasium: 0.28.1
Checklist
[ ] My issue does not relate to a custom gym environment. (Use the custom gym env template instead)
[X] I have checked that there is no similar issue in the repo
🐛 Bug
I am running the learn.py example that came with the repo, no modifications.
It seems like the training works perfectly fine and also saves the model, but I encounter an error at that instance of predict() method. It seems that there are inconsistencies with what gym vs sb3 returns for the reset() method, hence inputting an invalid argument into predict(). I tried this bypass, and while it fixed the issue at predict(), it ran into another issue later on running evaluation.py in the stable_baselines3 package, causing me to be unable to evaluate my state. This other issue is of the same nature as it has another problem with the dimensions that are returned with reset().
I made sure that I have gymnasium support installed as well. I'm kind of at a loss for what to do here. Does anyone know if I am supposed to edit the source code in sb3?
To Reproduce
Relevant log output / Error message
System Info
Checklist