Ball/paddle positions off in the real frame

[akanksha-atrey]

I have been trying to map pixels to the objects (i.e. ball, paddle, bricks) in breakout. But, for some reason, the ball position in the state json is always very off and paddle positions are sometimes off. I am not sure if this is a bug in my code or if I am missing something about the nature of Atari (i.e. jittering, etc?).

Here are 6 consecutive frames where white are the pixels generated from my code based on the ball and paddle positions:

Note: in all of the above images, there is no overlap between the ball position pixels suggesting that ball positions are consistently off.

[kclary]

Reposting code @akanksha95 shared in the slack to generate these images:

obs = env.reset()turtle = atari_wrappers.get_turtle(env)
    tb = turtle.toybox
    start_state_json = tb.state_to_json()
    history['state_json'].append(start_state_json)
    num_steps = -1

  while not done and episode_length <= max_ep_len:
        episode_length += 1
        actions, value, _, _, a_logits = model.step(obs)
        num_lives = turtle.ale.lives()
        obs, reward, done, info = env.step(actions)
        state_json = tb.state_to_json()
        color_frame = tb.get_rgb_frame()

        #save info        
        history['state_json'].append(state_json)
        history['color_frame'].append(color_frame)

Where/when do you save the image? This might be related to the fact that a stack of four images is used to select the next action, or that some frames are skipped for efficiency.

[akanksha-atrey]

@kclary I added in the above code where I am saving the image. It is right when I save the state JSON. To add to this, this is the partial code I am using to get the pixels for each object (shown for the ball object here):

pixels = []
for i in range(30, len(history['state_json'])):
        frame = history['color_frame'][i]
        state_json = history['state_json'][i]
        ball_pos = (int(state_json[concept][0]['position']['x']), int(state_json[concept][0]['position']['y']))
        ball_radius = int(state_json['ball_radius'])
        pixels += [ball_pos]
        ...do something with position and radius to get all the pixels that a ball covers...

        #change pixels to white to see mapping in the real frame
        for pixel in concept_pixels:
             frame[pixel[1], pixel[0], 0] = 255
             frame[pixel[1], pixel[0], 1] = 255
             frame[pixel[1], pixel[0], 2] = 255
        plt.imshow(frame)
        plt.show()

[kclary]

Could you include all of the code you're using to white out the concept pixels? Or add it to the other repo?

[akanksha-atrey]

I pushed it to the other repo! Note, it is not complete but the preliminary logic is there. It is located in saliency_maps/experiments/CF_importance.py.

[kclary]

Have you seen this error in each of the concepts, or just ball and paddle?

[akanksha-atrey]

I have only looked at paddle and ball positions so far. Working on the brick pixels now but I imagine the same problem wouldn't exist since brick locations are static in a game.

[akanksha-atrey]

Just to confirm, the issue does not exist with brick locations.

[jjfiv]

In the loop, you are doing:

history['state_json'].append(state_json)
history['color_frame'].append(color_frame)

So I'm assuming you've got the two lists in lockstep. Can you check they're the same size? In the code you posted here, it looks like you add a start state_json but not a start frame, so this may just be an off-by-one?

[kclary]

Just verified that rollout.py in the saliency_maps repo (https://github.com/KDL-umass/saliency_maps/blob/master/visualize_atari/rollout.py) does add to history['state_json'] but not history['color_frame'] outside of the loop, which would mean indexes to state_json would not align with the other arrays in history.

[akanksha-atrey]

Thanks for noticing that Kaleigh! I just noticed it as well. I don't see the same thing in older commits and this was probably a recent bug in the code. I will fix this and verify whether the position error is gone. Also, will add some tests to ensure correctness!

[akanksha-atrey]

Just verified, that did fix the position error! This leads to another question. I was mapping the location in the state_json to the frame and this is what I got (see white dot on ball and paddle):

It appears that the paddle position is always the mid-top. However, the ball position seems like it is oddly located near the bottom left corner, but not quite at a distinguishable location. Do you know why that is?

[jjfiv]

Yes! The ball position corresponds to the middle of the ball. When you draw a pixel at it, it looks offset like that, but pixel drawing is top-left-corner. This was chosen because the ball hits things in all four directions.

The paddle is represented by the top, because its clear that we only care about the top edge of the paddle.

@akanksha95: if you're all set with this issue, go ahead and close.

[akanksha-atrey]

Awesome, thanks for the help and info!