Bug of PPO

[GIS-PuppetMaster]

ratio = tf.exp(pi.log_prob(action) - old_pi.log_prob(action)) surr = ratio adv ... loss = -tf.reduce_mean( tf.minimum(surr, tf.clip_by_value(ratio, 1. - self.epsilon, 1. + self.epsilon) adv) )

should use ratio in tf.minimum rather than surr, because surr=ration*adv, and there could be negative value in adv, so the result of tf.minimum may contain a value like -1e10, and cause actor's loss failed.

[GIS-PuppetMaster]

it should be like this: self.cliped_ratio = tf.clip_by_value(self.ratio, 1. - METHOD['epsilon'],

1. • METHOD['epsilon']) self.min_temp = tf.minimum(self.ratio, self.cliped_ratio) self.aloss = -tf.reduce_mean(self.min_temp * self.tfadv)

[quantumiracle]

Why the negative value causes failure in actor loss? You can also refer to OpenAI baselines here, which has similar process as our repo.

[GIS-PuppetMaster]

Why the negative value causes failure in actor loss? You can also refer to OpenAI baselines here, which has similar process as our repo.

I drawed the loss polt and reward plot, when there is a very small negative value, such as 1e-10, the loss will be extremly larger than normally, and the reward stoped increase. I just tried lower learning rate, and there was no such 1e-10 value came out. I wonder if it's the same that use my code above, since it's more robust.

[quantumiracle]

Sorry for the late reply. What you mentioned might be caused by some numerical issues in tf.minimum if I understood correctly. Could you please print out an example case and paste it here? I'm a bit confused by your description since you mentioned both large negative value (-1e10) and small positive value (1e-10). A case showing how it causes a large loss value would be great.