Computing world pose for extreme pixel coordinates

[rimvydasrub]

when running the following code:

from mujoco_robot_environments.tasks.rearrangement import RearrangementEnv
import numpy as np

env = RearrangementEnv()
env.reset()

min_coords = np.array([0.0, 0.0])
max_coords = np.array([env.overhead_camera_width, env.overhead_camera_height])

# get a word coordinates
min_pose = env.pixel_2_world("overhead_camera", min_coords)
max_pose = env.pixel_2_world("overhead_camera", max_coords)

print(f"min_coords: {min_coords} max_coords: {max_coords}")
print(f"min_pose: {min_pose} max_pose: {max_pose}")

the following error occurs:

  File "/mujoco_robot_environments/test_coords.py", line 12, in <module>
    max_pose = env.pixel_2_world("overhead_camera", max_coords)
  File "/mujoco_robot_environments/mujoco_robot_environments/tasks/rearrangement.py", line 526, in pixel_2_world
    depth_val = depth_vals[coords_rounded[1], coords_rounded[0]]
IndexError: index 480 is out of bounds for axis 0 with size 480

this edge case is important if one would like to dynamically calculate the regions for place location

[peterdavidfagan]

This is due to zero indexing, the image height and width values should be out of bounds for the image arrays.

[rimvydasrub]

The question is what is the appropriate way to handle this. There are two options:

1. change the API so as to handle such values
2. change the ranges sith the correction like this:

   min_coords = np.array([0.0, 0.0])
   max_coords = np.array([env.overhead_camera_width-1, env.overhead_camera_height-])

Its fine either way, but it should be clear what the approach moving forward

[rimvydasrub]

IMO: its faster with 2, but in future you might want to revisit.