Closed yangyiqin-tsinghua closed 4 years ago
In the TRPO paper, the Hessian of the KL is computed at \theta = \theta_{old}. So the two action probs are the same in value, but the action_prob0 representing \pi_old is detached so that no gradient will flow from it.
Thank you very much!
Thank you very much!
Shouldn't kl be two different strategies? There action_prob1 == action_prob0?? Thank you