First of all thanks for your great tutorial on reinforcement learning, I went through all the series and learned a lot.
In the PPOAgent I think there may be something wrong with this line. When I vstack the discounted_r (shape of (n,1)) and subtract it with predicted values (shape of (n,)), the advantages become shape of (n,n). So I think maybe we should not vstack discounted_r, but vstack the advantages in this lineadvantages = np.vstack(discounted_r - values), then the advantages are shape of (n,1), which is the expected result.
Hi, Rokas:
First of all thanks for your great tutorial on reinforcement learning, I went through all the series and learned a lot.
In the PPOAgent I think there may be something wrong with this line. When I
vstackthe discounted_r (shape of (n,1)) and subtract it with predicted values (shape of (n,)), the advantages become shape of (n,n). So I think maybe we should notvstackdiscounted_r, butvstackthe advantages in this lineadvantages = np.vstack(discounted_r - values), then the advantages are shape of (n,1), which is the expected result.Thanks.