search

LyWangPX / Reinforcement-Learning-2nd-Edition-by-Sutton-Exercise-Solutions

Solutions of Reinforcement Learning, An Introduction
MIT License
2.04k stars 465 forks source link

[Ex 4.2] Changing dynamics changes the state values #85

Open Jonathan2021 opened 3 years ago

Jonathan2021 commented 3 years ago

The state 15 (with state 13's dynamics changed), isn't equivalent to state 13. It is further away from the upper left terminal state but closer to lower right (left, right and down are equivalent to state 13, but up makes it closer to lower right than up in state 13). I ran your script 4.2.py (by the way going left and right in state 15 leads to 12 and 14 respectively and not to state 15 as written in your script), added a print in the draw function for state 15 and you can see that the decimals are not the same as for state 13. You have to recalculate the whole game. All the states changed slightly (those further away changed less) if you take the decimals into account (compared to running your script for 4.1 which by the way prints value-1 in the board for some weird reason but the accurate state value list is ok). Thanks for your efforts in providing a correction for the exercises !

Arpitrf commented 3 years ago

So, we cannot give the answer to this part of the question by hand, right? (i.e. to get the new state values, we'll have to program it)?