calico-1226
commented
1 year ago In the implementation of the IPO paper, we encountered two issues with the logarithmic form. Firstly, because reinforcement learning estimates the sum of costs from sampling to obtain the expected policy, even starting from a feasible policy, the optimization process often encounters situations where the sum of costs exceeds a threshold. This causes the numerical value in the logarithm to be less than 0, making the IPO algorithm unable to run. Secondly, when the sum of costs is very close to the threshold, the problem of gradient explosion arises due to the pathological curvature in gradient descent caused by the Hessian matrix with a large condition number.
Therefore, we made certain equivalent transformations to the theory of IPO. In situations where the constraints are violated and the gradient is too large, we replaced the gradient with an artificially given maximum value. The specific theoretical analysis is as follows.
In IPO, the loss function is
whose gradient is
Thus, we rewrite the loss function as follows,
where the parameter $\lambda_{i,\theta}$ is defined as the minimum of $\lambdai(\theta)$ and a maximum value $\lambda\text{max}$ to avoid issues with ill-conditioning. It is a parameter related to $\theta$ but detached when calculating the gradient flow. Notably, $\mathcal L^\lambda(\theta)$ and $\mathcal L(\theta)$ have the same gradient when $\lambda_{i,\theta}=\lambdai(\theta)$. Moreover, when IPO violates constraints, we can still update the policy using $L^\lambda(\theta)$ by simply setting $\lambda{i,\theta}=\lambda_\text{max}$.
zmsn-2077
stvsd1314
Xi-HHHM
Gaiejj
Andrewllab
Required prerequisites
Questions
About the IPO algorithm, why the form is not the same as original paper(the log form)? It makes me confused. And by the way, I find that the effect of using IPO is bad than simply using penalty cost, what is the reason? Thanks !