book errata p 682

[elfelround]

book errata p 682

[vmirly]

Thanks for pointing out this errata. You are right, I think I missed counting the final step. At t=7 an action is taken in order to go to the terminal state T.

So for episode 1, t goes from 0 to 8, and recall that the sequence is denoted by

• t=0: <S0, A0, R1>
• t=1: <S1, A1, R2>
• t=2: <S2, A2, R3>
• ..
• t=7: <S7, A7, R8>
• Terminal state T

Also , we know the following immediate rewards: R1=0, R2=0, ..., R6=0, R7=0, R8=1

So now let's calculate the returns for the first episode:

• At t=0 : S_0=B => G_0 = R1 + gamma R2 + ... + gamma^6 R7 + gamma^7 R8 = gamma^7 R8
• At t=1 : S_1=B => G_1 = R2 + gamma R3 + ... + gamma^5 R7 + gamma^6 R8 = gamma^6 R8
• At t=2 : S_2 =C => G_2 = R3 + gamma R4 + ... + gamma^4 R7 + gamma^5 R8 = gamma^5 R8
• At t=3 : S_3 =C => G_3 = R4 + gamma R5 + ... + gamma^3 R7 + gamma^4 R8 = gamma^4 R8
• At t=4 : S_4 =C => G_4 = R5 + gamma R6 + gamma^2 R7 + gamma^3 R8 = gamma^3 R8
• At t=5 : S_5 =C => G_5 = R6 + gamma R7 + gamma^2R8= gamma^2 * R7
• At t=6 : S_6 = B => G_6 = R7 + gamma * R8 = gamma
• At t=7 : S_7 = A => G_7 = R8 = 1
• Terminal state T

Similarly, for episode 2, t goes from 0 to 10, and R0=R1=...=R9=0 while R10=-1

• At t=0 : S_0=A => G_0 = R1 + gamma R2 + ... + gamma^9 R10 = gamma^9 * R10
• At t=1 : S_1=B => G_1 = R2 + gamma R3 + ... + gamma^8 R10 = gamma^8 * R10
• At t=2 : S_2 =B => G_2 = R3 + gamma R4 + ... + gamma^7 R10 = gamma^7 * R10
• At t=3 : S_3 =B => G_3 = R4 + gamma R5 + ... + gamma^6 R10 = gamma^6 * R10
• At t=4 : S_4 =B => G_4 = R5 + gamma R6 + ... + gamma^5 R10 = gamma^5 * R10
• At t=5 : S_5 =B => G_5 = R6 + gamma R7 + ... + gamma^4 R10 = gamma^4 * R10
• At t=6 : S_6 =B => G_6 = R7 + gamma R8 + ... + gamma^3R10 = gamma^3 * R10
• At t=7 : S_7 =B => G_7 = R8 + gamma R9 + gamma^2 R10 = gamma^2 * R10
• At t=8 : S_8 =B => G_8 = R9 + gamma R10 = gamma R10
• At t=9 : S_9 = B => G_9 = R10 = -1
• Terminal state T

[elfelround]

@vmirly posted this errata via packt but it was annoying as heck to explain without an image, they just said, can u further explain? and i thought fuck it. having a corrected answer will facilitate my understanding of this part, ill read it when i have time with my current book and let you know how this follows :) also the RL chapter is still great, but feels rushed in comparison with whole book, maybe a bit more love to it on 4th ed? xx

[rasbt]

also the RL chapter is still great, but feels rushed in comparison with whole book, maybe a bit more love to it on 4th ed? xx

Oh yeah for sure. The rewrites for Tf 2.0 took much longer than expected. And we both were very busy in Fall (due to my teaching responsibilities and Vahid starting a new position). It definitely could and should be smoothened out in a potential next edition. Thanks for your feedback!