chapter04/gamblers_problem.py line33 to 62 may has a problem

[ChenHuaYou]

while True:
    old_state_value = state_value.copy()
    sweeps_history.append(old_state_value)

    for state in STATES[1:GOAL]:
        # get possilbe actions for current state
        actions = np.arange(min(state, GOAL - state) + 1)
        action_returns = []
        for action in actions:
            action_returns.append(
                HEAD_PROB * state_value[state + action] + (1 - HEAD_PROB) * state_value[state - action])
        new_value = np.max(action_returns)
        state_value[state] = new_value
    delta = abs(state_value - old_state_value).max()
    if delta < 1e-9:
        sweeps_history.append(state_value)
        break

# compute the optimal policy
policy = np.zeros(GOAL + 1)
for state in STATES[1:GOAL]:
    actions = np.arange(min(state, GOAL - state) + 1)
    action_returns = []
    for action in actions:
        action_returns.append(
            HEAD_PROB * state_value[state + action] + (1 - HEAD_PROB) * state_value[state - action])

    # round to resemble the figure in the book, see
    # https://github.com/ShangtongZhang/reinforcement-learning-an-introduction/issues/83
    policy[state] = actions[np.argmax(np.round(action_returns[1:], 5)) + 1]

i think from the line # compute the optimal policy to the end of for loop, it should be in the while loop

[ChenHuaYou]

but i put the policy evaluate and the policy improvement in the bigger while loop , found it run twice evaluate and once improvement makes the result equivalent to your result .. , i dont know if it is a coincidence

[ShangtongZhang]

It's not a bug. It's value iteration not policy iteration, so there is no policy evaluation. Computing the optimal policy is solely for plotting.