Hyperparameter tuning using Custom env with RL-zoo and SB2

[shivam-saurav]

Important Note: We do not do technical support, nor consulting and don't answer personal questions per email.

If you have any questions, feel free to create an issue with the tag [question].
If you wish to suggest an enhancement or feature request, add the tag [feature request].
If you are submitting a bug report, please fill in the following details.

If your issue is related to a custom gym environment, please check it first using:

from stable_baselines.common.env_checker import check_env

env = CustomEnv(arg1, ...)
# It will check your custom environment and output additional warnings if needed
check_env(env)

Describe the bug A clear and concise description of what the bug is.

Code example Please try to provide a minimal example to reproduce the bug. Error messages and stack traces are also helpful.

Please use the markdown code blocks for both code and stack traces.

from stable_baselines import ...

Traceback (most recent call last): File ...

System Info Describe the characteristic of your environment:

• Describe how stable baselines was installed (pip, docker, source, ...)
• GPU models and configuration
• Python version
• Tensorflow version
• Gym version
• Pybullet version
• Versions of any other relevant libraries

Additional context Add any other context about the problem here.

[araffin]

as a quick remark, please use SB3 and the associated RL zoo (cf readme), as SB2 is no longer actively developped. And please fill the issue template properly.

[shivam-saurav]

I am trying to tune the hyperparameters but the error says : Trial 2 failed because of the following error: AttributeError("'A2C' object has no attribute 'env'").

However, I have checked with check_env(),it works well with that.

check_env(env, warn=True) (It works fine) Output for this: /usr/local/lib/python3.7/dist-packages/gym/logger.py:30: UserWarning:

WARN: Box bound precision lowered by casting to float32

Instantiate the agent using : model = A2C('MlpPolicy', env, verbose=1)

And trying to tune the hyperparameter using :

hyperparams={} data_frame = hyperparam_optimization('a2c', model, model.get_env(), n_trials=100, n_timesteps=5000, hyperparams=hyperparams, n_jobs=2, seed=0, sampler_method='random', pruner_method='median', verbose=1)

The error is :

[I 2021-05-14 06:39:38,453] Trial 1 finished with value: 800.7374267578125 and parameters: {'gamma': 0.995, 'n_steps': 32, 'lr_schedule': 'constant', 'lr': 0.00026730495810056736, 'ent_coef': 1.0750232421652547e-05, 'vf_coef': 0.3385992446532201}. Best is trial 1 with value: 800.7374267578125. [W 2021-05-14 06:39:38,469] Trial 2 failed because of the following error: AttributeError("'A2C' object has no attribute 'env'") Traceback (most recent call last): File "/usr/local/lib/python3.7/dist-packages/optuna/_optimize.py", line 217, in run_trial value_or_values = func(trial) File "", line 104, in objective eval_freq = max(eval_freq // model.get_env().num_envs, 1) File "/usr/local/lib/python3.7/dist-packages/stable_baselines/common/base_class.py", line 104, in get_env return self.env

AttributeError: 'A2C' object has no attribute 'env'

Just to clarify few things, I used the hyperparam_optimization function(from https://github.com/araffin/rl-baselines-zoo/blob/master/utils/hyperparams_opt.py) with few changes in the objective function.

Line 98 was changed to : model = model_fn #from model = model_fn(**kwargs) Line 100 was changed to : evalfreq = eval_freq #from env_fn(n_envs=1, eval_env=True)

I understand that rl-baselines-zoo can be installed and required command can be run but that too requires the gym registry of the custom-env which was creating issue. I just want to clarify what's wrong in doing this and why the error is 'A2C' object has no attribute 'env' when A2C atually has.

[araffin]

Just to clarify few things, I used the hyperparam_optimization function(from https://github.com/araffin/rl-baselines-zoo/blob/master/utils/hyperparams_opt.py) with few changes in the objective function.

then please check that the script works without your modification and a normal env. If so, then you know where the issue come from ;) (we don't do tech support as mentioned explicitly in the issue template and README)

but that too requires the gym registry of the custom-env which was creating issue.

why is that an issue? https://github.com/DLR-RM/rl-baselines3-zoo#custom-environment

PS: please use markdown to format your code and traceback (cf issue template)