Implementation of some expectation is different from paper

[XinyuWang2]

Hi, I find some expectation is not calculated the way written in the paper. I know some expectations are impossible to calculate because of infinite possibilities. Did you try a sampling method and compare it with the way implemented? I am really curious about the difference.

E.g. When calulating A,

• In the paper: A = E_sg ~ piH (C(sg | s, g)), so the probability of piH is applied on C.
• implementation: It first calculated the expectation of sg (sg_hat), and let A = C(sg_hat | s, g)

[elliotchanesane31]

Hi, you are right, there is a small difference between the paper derivation and the actual implementation to compute A.

In the paper, the advantage A(sg | s, g) = E_{sg_hat ~ piH(.|s, g)} [C(sg_hat|s, g)] – C(sg |s, g) is the difference of two terms, the first being an expectation with respect to piH(.|s, g). In principle, we should average C(sg_hat |s, g) for multiple subgoals sg_hat sampled from piH(.|s, g) to approximate this expectation, which could be computationally expensive depending on the number of samples we choose.

In practice, we found that using only the mean of piH(.|s, g) for sg_hat was simpler and faster as well as more stable than a sampling based approximation, although I haven’t looked into this in depth.