search

rlworkgroup / garage

A toolkit for reproducible reinforcement learning research.
MIT License
1.86k stars 310 forks source link

Multiple gradient steps in Meta-RL evaluation adaptation #2309

Closed benedikt-schesch closed 1 year ago

benedikt-schesch commented 2 years ago

Hello,

I am trying to run garage/examples/torch/maml_ppo_half_cheetah_dir.py and I would like to adapt the number of gradient steps that the algorithm takes for adaptation to a new task. Is there a proper way to do this because I haven't found anything?

Thank you very much

Benedikt

krzentner commented 2 years ago

Hi Benedikt.

Unfortunately there isn't currently any way of doing that with the MetaEvaluator. A pull request to implement it would be appreciated.

There is currently the num_grad_updates in MAML, but it's only used during training, not evaluation. Theoretically, the logged training results for MAML should be the same as meta-evaluating, assuming that your meta-task has no test / train split.

The other implemented meta-RL algorithms (RL^2 and PEARL) don't have an obvious interpretation of "multiple gradient steps," since they don't use gradient steps at meta-test time.