Closed BryanZ666 closed 1 year ago
Indeed - StableBaselines 3 internally transposes the env to be channels-first for training, but rollout.rollout
is using the env that you passed in. The solution is to instead call
rollouts = rollout.rollout(
expert,
expert.get_env(),
rollout.make_sample_until(min_timesteps=None, min_episodes=2),
)
Then you get a different error that I don't really understand - unwrap_traj
ends up calling the 'rollout'
key of some object that doesn't have that key.
At any rate, we should probably document this issue of rolling out on the wrong env somewhere.
Thanks for your comments, is there anything I could try to overcome this problem? what if I set the unwrap to False to bypass the unwrap_traj function? btw, I simply changed the expert to be PPO but the agent seems could not learn from the environment, what could be the problem?
Then you get a different error that I don't really understand -
unwrap_traj
ends up calling the'rollout'
key of some object that doesn't have that key.
https://github.com/HumanCompatibleAI/imitation/blob/master/src/imitation/data/wrappers.py#L160 adds the "rollout" key. I do not know why that would be getting lost in algorithm.get_env()
-- I guess one of the wrappers SB3 applise might eliminate the info dict, though that seems counterintuitive.
Setting unwrap
to False is probably a viable workaround. It does mean you'll lose the last observation (VecEnv
discards it as it auto-resets), but that might be OK for your use case.
Yeah, now it can rollouts trajectories! But the agent could learn meaningful policy using the default BC parameter.
bc_trainer = bc.BC(
observation_space=env.observation_space,
action_space=env.action_space,
demonstrations=transitions,)
bc_trainer.train(n_epochs=300)
is there anything I could try to improve it?
To sanity check, the transitions you're training BC on are transposed correctly relative to the environment you're giving BC? I imagine it would error out if not, but if it was treating channel dimension as batch dimension that would certainly prevent it from learning...
Otherwise, I don't have much to advise other than "tune hyperparameters". Behavioral cloning is a pretty weak imitation learning algorithm anyway, so it's common for it to require methods like data augmentation. Something like GAIL tends to be a lot more robust.
Closing due to inactivity.
Bug description
I try to rollout the trajectories by PPO/A2C for task "BreakoutNoFrameskip-v4". An error occurred "ValueError: Observation spaces do not match". I think the Observation got transposed, (4,84,84) !=(84,84,4).
Steps to reproduce
Environment
pip freeze --all
: