We now provide methods for constructing the clipped MPO (CMPO) policy targets used as part of the Muesli agent loss. These CMPO targets are in expectation proportional to: prior(a|s) * exp(clip(norm(Q(s, a)))) where the prior is computed by the actor policy head, and the Q values are computed using the learned model's reward and value heads.
rlax: Upstream Muesli utilities to rlax.
We now provide methods for constructing the clipped MPO (CMPO) policy targets used as part of the Muesli agent loss. These CMPO targets are in expectation proportional to:
prior(a|s) * exp(clip(norm(Q(s, a))))where the prior is computed by the actor policy head, and the Q values are computed using the learned model's reward and value heads.See "Muesli: Combining Improvements in Policy Optimization" by Hessel et al. (https://arxiv.org/pdf/2104.06159.pdf) for more details.