search

DLR-RM / rl-baselines3-zoo

A training framework for Stable Baselines3 reinforcement learning agents, with hyperparameter optimization and pre-trained agents included.
https://rl-baselines3-zoo.readthedocs.io
MIT License
1.89k stars 494 forks source link

Issue with 'feat/offline-RL' Branch for Donkey Car in rl-baselines3-zoo #419

Open Gribesh opened 7 months ago

Gribesh commented 7 months ago

❓ Question

Hello everyone,

I am following @araffin video Part 1 for donkey car https://www.youtube.com/watch?v=ngK33h00iBE

I'm currently working on a project using the rl-baselines3-zoo with the aim of training a reinforcement learning model for the Donkey Car environment. However, I've encountered a challenge with different branches in the repository.

When I run the training command on the feat/offline-RL branch:

$ python train.py --algo tqc --env donkey-waveshare-v0 --eval-freq -1 --save-freq 10000

I get the following error (gymnasium.error.Error: No registered env with id: donkey-waveshare-v0.):

========== donkey-waveshare-v0 ==========
Seed: 1107839033
Default hyperparameters for environment (ones being tuned will be overridden):
OrderedDict([('batch_size', 256),
             ('buffer_size', 200000),
             ('callback',
              [{'utils.callbacks.ParallelTrainCallback': {'gradient_steps': 200}}]),
             ('ent_coef', 'auto'),
             ('gamma', 0.99),
             ('learning_rate', 0.00073),
             ('n_timesteps', 2000000.0),
             ('policy', 'CnnPolicy'),
             ('tau', 0.02),
             ('train_freq', 200)])
Using 1 environments
Traceback (most recent call last):
  File "racing\donkey\rl-baselines3-zoo\train.py", line 220, in <module>
    model = exp_manager.setup_experiment()
  File "racing\donkey\rl-baselines3-zoo\utils\exp_manager.py", line 173, in setup_experiment
    env = self.create_envs(n_envs, no_log=False)
  File "racing\donkey\rl-baselines3-zoo\utils\exp_manager.py", line 522, in create_envs
    env = make_vec_env(
  File "racing\.venv\lib\site-packages\stable_baselines3\common\env_util.py", line 125, in make_vec_env
    vec_env = vec_env_cls([make_env(i + start_index) for i in range(n_envs)], **vec_env_kwargs)
  File "racing\.venv\lib\site-packages\stable_baselines3\common\vec_env\dummy_vec_env.py", line 30, in __init__
    self.envs = [_patch_env(fn()) for fn in env_fns]
  File "racing\.venv\lib\site-packages\stable_baselines3\common\vec_env\dummy_vec_env.py", line 30, in <listcomp>
    self.envs = [_patch_env(fn()) for fn in env_fns]
  File "racing\.venv\lib\site-packages\stable_baselines3\common\env_util.py", line 94, in _init
    env = gym.make(env_id, **kwargs)  # type: ignore[arg-type]
  File "racing\.venv\lib\site-packages\gymnasium\envs\registration.py", line 741, in make
    env_spec = _find_spec(id)
  File "racing\.venv\lib\site-packages\gymnasium\envs\registration.py", line 528, in _find_spec
    raise error.Error(
gymnasium.error.Error: No registered env with id: donkey-waveshare-v0. Did you register it, or import the package that registers it? Use `gymnasium.pprint_registry()` to see all of the registe
red environments.

I've made sure that my environment is correctly named and registered. I am able run from interactive python. Despite this, the error persists, which prevents the script from finding the donkey-waveshare-v0 environment within Gymnasium's registry.

On the other hand, when I switch to the feat/gym-donkeycar branch, the environment loads without any issues, and the training begins as expected.

========== donkey-waveshare-v0 ==========
Seed: 1720947574
Loading hyperparameters from: racing\donkey\rl-baselines3-zoo\hyperparams\tqc.yml
Default hyperparameters for environment (ones being tuned will be overridden):
OrderedDict([('batch_size', '256p'),
             ('buffer_size', 20000),
             ('callback',
              [{'rl_zoo3.callbacks.ParallelTrainCallback': {'gradient_steps': 200}}]),
             ('ent_coef', 'auto'),
             ('gamma', 0.99),
             ('gradient_steps', 256),
             ('learning_rate', 0.00073),
             ('learning_starts', 20000),
             ('n_timesteps', 3000000.0),
             ('policy', 'CnnPolicy'),
             ('policy_kwargs',
              'dict(log_std_init=-3, net_arch=[256, 256], n_critics=2, '
              'use_expln=True)'),
             ('sde_sample_freq', 16),
             ('tau', 0.02),
             ('train_freq', 200),
             ('use_sde', True),
             ('use_sde_at_warmup', True)])
Using 1 environments
starting DonkeyGym env
Setting default: start_delay 5.0
Setting default: max_cte 8.0
Setting default: frame_skip 1
Setting default: cam_resolution (120, 160, 3)
Setting default: log_level 20
Setting default: host localhost
Setting default: port 9091
Setting default: steer_limit 0.5
Setting default: throttle_min 0.0
Setting default: throttle_max 1.0
INFO:gym_donkeycar.core.client:connecting to localhost:9091
racing\.venv\lib\site-packages\gym\spaces\box.py:78: UserWarning: WARN: Box bound precision lowered by casting to float32
  logger.warn(f"Box bound precision lowered by casting to {self.dtype}")
loading scene waveshare
WARNING:gym_donkeycar.envs.donkey_sim:waiting for sim to start..
INFO:gym_donkeycar.envs.donkey_sim:on need car config
INFO:gym_donkeycar.envs.donkey_sim:sending car config.
INFO:gym_donkeycar.envs.donkey_sim:sim started!
racing\.venv\lib\site-packages\stable_baselines3\common\vec_env\patch_gym.py:49: UserWarning: You provided an OpenAI Gym environment. We strongly recommend transitioning to
 Gymnasium environments. Stable-Baselines3 is automatically wrapping your environments in a compatibility layer, which could potentially cause issues.
  warnings.warn(
Wrapping the env in a VecTransposeImage.
Using cpu device
Log path: logs/tqc/donkey-waveshare-v0_3
Saving to logs/tqc/donkey-waveshare-v0_3

Could someone help me understand why the feat/offline-RL branch does not recognize the environment while the feat/gym-donkeycar branch does? Is there a step I might be missing in the setup for the feat/offline-RL branch?

Additionally, I'm curious to know if there is a more recent branch or version of rl-baselines3-zoo that is recommended for use with Donkey Car simulations. If there is a newer branch, could you please direct me to it?

Any insights or assistance would be greatly appreciated!

Checklist

araffin commented 7 months ago

Hello, the offlineRL branch is probably outdated. The last branch that I tested was https://github.com/DLR-RM/rl-baselines3-zoo/pull/260 (but I haven't tested it since one year)

Could someone help me understand why the feat/offline-RL branch does not recognize the environment while the feat/gym-donkeycar branch does?

This is the same issue as https://github.com/araffin/aae-train-donkeycar/issues/7#issuecomment-1794379988 There was a migration from gym to gymnasium, and the feat/gym-donkeycar does offer some patching to help with the migration (see warnings in the terminal). A better solution would be to migration gym donkey car repo to gymnasium.

Gribesh commented 7 months ago

Thanks for the update https://github.com/DLR-RM/rl-baselines3-zoo/pull/260 does work, I had to make few changes to run enjoy.py file to make it run.

Also can you let me know what parameter I can give while running enjoy.py file so that it can run in real donkey car (like: JetRacer) as well?

araffin commented 7 months ago

Also can you let me know what parameter I can give while running enjoy.py file so that it can run in real donkey car (like: JetRacer) as well?

I've only tried with my custom car so far. This is more related to gym env you are using that will interface with the car, the RL Zoo/SB3 only interact via the gym API (reset(), step()).

Gribesh commented 7 months ago

Thanks for the confirmation. I was looking at the parameters and making tweaks.

Hopefully, I can use donkey car manage.py with the trained model using rl baseline3 zoo. (As I am using waveshare so environment might be similar enough to make some actions in real world )