search

rlworkgroup / garage

A toolkit for reproducible reinforcement learning research.
MIT License
1.84k stars 309 forks source link

Suggestion to add how to implement pre-trained policies. #2325

Open bara-bba opened 2 years ago

bara-bba commented 2 years ago

Hi! I'm currently working to implement a policy trained using SAC in mujoco into a real robot. I'm trying to load the two q-functions but I obtain weird result in q_loss and the returns. Any suggestion in how to load correctly the policy? Thanks!

krzentner commented 2 years ago

I would recommend comparing the observation distribution between your real-world environment and simulated environment. The most obvious difficulties in sim2real transfer are due to that mismatch. Note that the docs already describe how to use a pre-trained policy. Continuing to train after transferring from sim2real is an active area of research, and I don't have a firm recommendation for how to achieve it. In particular, a Q function describing a policy's behavior in simulation is likely to over-estimate the performance of that policy on the real environment.