search

DLR-RM / rl-baselines3-zoo

A training framework for Stable Baselines3 reinforcement learning agents, with hyperparameter optimization and pre-trained agents included.
https://rl-baselines3-zoo.readthedocs.io
MIT License
1.9k stars 496 forks source link

[Feature request] Hyperparameter optimization from pretrained agent #91

Open Jonathan2021 opened 3 years ago

Jonathan2021 commented 3 years ago

Enabling the possibility to run --optimize with the --trained-agent flag would be great ! In my case, I pre-trained an agent on a simplified task and want to continue training it on the real task (which involves a modified reward, more obstacles etc.). It would be great to be able to run a hyperparameter search for this second phase of the training. (Even though some hyperparameters, such as the network architecture, can't be tuned here). For now, when I run both flags together, it just continues training (weirdly outputting less info than without the --optimize flag by the way). Thanks for the awesome training framework !

araffin commented 3 years ago

Hello, I'm unsure about such feature. On one side, it seems to be a reasonable (even though unconventional) request. On the other side there are some behaviors that may be ill-defined. For instance, if your pre-trained agent has a replay buffer size of 1e6, you should not change that during hyperparameter optimization. The same goes with other hyperparameters.

As a compromise you can fork this repo and create the feature in it (and post a link there ;)).

Jonathan2021 commented 3 years ago

Not sure I will have enough time nor the courage to do so but maybe. Thanks for the reply anyways ;)