Error when run hyperparameter optimization

blurLake commented 4 years ago

Hi, I was trying the following code from README python train.py --algo ppo --env MountainCar-v0 -n 50000 -optimize --n-trials 1000 --n-jobs 2 \ --sampler random --pruner median, but I got the following error. Could you tell me where the problem possibly is? Thanks!

Error message

Seed: 1990554247
OrderedDict([('ent_coef', 0.0),
             ('gae_lambda', 0.98),
             ('gamma', 0.99),
             ('n_envs', 16),
             ('n_epochs', 4),
             ('n_steps', 16),
             ('n_timesteps', 1000000.0),
             ('nminibatches', 1),
             ('normalize', True),
             ('policy', 'MlpPolicy')])
Using 16 environments
Overwriting n_timesteps with n=50000
Normalizing input and reward
Optimizing hyperparameters
Sampler: tpe - Pruner: median
Normalizing input and reward
[W 2020-06-25 11:09:18,251] Setting status of trial#0 as TrialState.FAIL because of the following error: TypeError("__init__() got an unexpected keyword argument 'nminibatches'")
Traceback (most recent call last):
  File "/Users/eejiiew/Projects/filter-zoo/venv/lib/python3.7/site-packages/optuna/study.py", line 734, in _run_trial
    result = func(trial)
  File "/Users/eejiiew/Projects/filter-zoo/rl-baselines3-zoo/utils/hyperparams_opt.py", line 84, in objective
    model = model_fn(**kwargs)
  File "train.py", line 385, in create_model
    verbose=0, **kwargs)
TypeError: __init__() got an unexpected keyword argument 'nminibatches'
Normalizing input and reward
[W 2020-06-25 11:09:18,346] Setting status of trial#1 as TrialState.FAIL because of the following error: TypeError("__init__() got an unexpected keyword argument 'nminibatches'")
Traceback (most recent call last):
  File "/Users/eejiiew/Projects/filter-zoo/venv/lib/python3.7/site-packages/optuna/study.py", line 734, in _run_trial
    result = func(trial)
  File "/Users/eejiiew/Projects/filter-zoo/rl-baselines3-zoo/utils/hyperparams_opt.py", line 84, in objective
    model = model_fn(**kwargs)
  File "train.py", line 385, in create_model
    verbose=0, **kwargs)
TypeError: __init__() got an unexpected keyword argument 'nminibatches'
Traceback (most recent call last):
  File "train.py", line 392, in <module>
    storage=args.storage, study_name=args.study_name, verbose=args.verbose)
  File "/Users/eejiiew/Projects/filter-zoo/rl-baselines3-zoo/utils/hyperparams_opt.py", line 119, in hyperparam_optimization
    study.optimize(objective, n_trials=n_trials, n_jobs=n_jobs)
  File "/Users/eejiiew/Projects/filter-zoo/venv/lib/python3.7/site-packages/optuna/study.py", line 382, in optimize
    for _ in _iter
  File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/joblib/parallel.py", line 934, in __call__
    self.retrieve()
  File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/joblib/parallel.py", line 833, in retrieve
    self._output.extend(job.get(timeout=self.timeout))
  File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
  File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/multiprocessing/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/joblib/_parallel_backends.py", line 567, in __call__
    return self.func(*args, **kwargs)
  File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/joblib/parallel.py", line 225, in __call__
    for func, args, kwargs in self.items]
  File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/joblib/parallel.py", line 225, in <listcomp>
    for func, args, kwargs in self.items]
  File "/Users/eejiiew/Projects/filter-zoo/venv/lib/python3.7/site-packages/optuna/study.py", line 648, in _reseed_and_optimize_sequential
    func, n_trials, timeout, catch, callbacks, gc_after_trial, time_start
  File "/Users/eejiiew/Projects/filter-zoo/venv/lib/python3.7/site-packages/optuna/study.py", line 682, in _optimize_sequential
    self._run_trial_and_callbacks(func, catch, callbacks, gc_after_trial)
  File "/Users/eejiiew/Projects/filter-zoo/venv/lib/python3.7/site-packages/optuna/study.py", line 713, in _run_trial_and_callbacks
    trial = self._run_trial(func, catch, gc_after_trial)
  File "/Users/eejiiew/Projects/filter-zoo/venv/lib/python3.7/site-packages/optuna/study.py", line 734, in _run_trial
    result = func(trial)
  File "/Users/eejiiew/Projects/filter-zoo/rl-baselines3-zoo/utils/hyperparams_opt.py", line 84, in objective
    model = model_fn(**kwargs)
  File "train.py", line 385, in create_model
    verbose=0, **kwargs)
TypeError: __init__() got an unexpected keyword argument 'nminibatches'

System Info

stable-baselines3-0.8.0a0

araffin commented 4 years ago

Could you tell me where the problem possibly is? Thanks!

This comes from https://github.com/DLR-RM/rl-baselines3-zoo/blob/master/hyperparams/ppo.yml#L50 You can just remove it, I forgot it from the rl-zoo with SB2.

If you have the time, you can also submit a PR to solve that issue ;)

blurLake commented 4 years ago

OK, I will do a test and submit one later.

DLR-RM / rl-baselines3-zoo

Error when run hyperparameter optimization #26