Closed wassname closed 7 years ago
Actually, in the latest version of your code I get all losses are positive, so you can ignore point 2.
Hello, thanks a lot for the interest!
log_std is the log of the standard deviation. The output of the model are mean, log_std and v, where the mean and the std are used to sample an action following a normal distribution. In that sense, yes, the std forces the exploration around the learned means. I don't know by reading the paper if this parameter must be learned. But learned or not, its value stays small (std stays close to 1) and the performance seems not affected.
Regarding the value loss, if you don't learn it, the advantage means nothing, so you can't learn the policy. In the paper, they don't say that it is not required when parameters are not shared, but they rather say that if parameters are shared, then we must train the value and the policy in the same gradient step (which makes sense). As far as I understand, in baseline, they don't share parameters in the default version, they just do it for the cnn version (for Atari games). Also, with my implementation, I found even lower performances when I share parameters (like in your suggested code).
That makes sense, thanks for the explanations!
I just went throught your code and the PPO paper and have a few questions, perhaps if you have time you could comment.
Should you take away lossvalue or add it? We want each individual loss to make the overall loss larger as they get larger. I can see you copied baselines exactly but maybe they have it wrong, In the PPO paper they have minus on one term (eq9). If you stop the training and inspect loss_clip and loss_value the first is negative, the second is positive. So it seems like we need to haveloss=loss_value-loss_clip
. Thoughts?