Open Boxxfish opened 1 year ago
On second thought, it would be more straightforward to write it like this:
let advantages = advantages.detach();
let ratio = (&action_log_probs - &old_action_log_probs).exp();
let term1 = &ratio * &advantages; // Importance sampling term
let ones = Tensor::ones(advantages.size(), (device, Kind::Float));
let term2 = Tensor::clip(&ratio, &ones - epsilon, &ones + epsilon) * &advantages; // Clipping term
let action_loss = -(term1.min(term2).mean(Kind::Float); // If importance sampling term doesn't fall between 1 - e to 1 + e, clip it
I noticed while browsing the RL examples that the PPO implementation is actually A2C (which there's already an example for). On line 141, this line:
should look something like this:
We should also be collecting old probabilities (
old_action_log_probs
) when collecting actions so we can perform the importance sampling.Also, and this is a nit, on line 151,
should be changed to
"ppo{update_index}.ot"
.