Obtain the real position of furnitures in the camera coordinate

[gaopeng5]

Thanks for your great work! Following the question on setting multiple cameras, once I set different cameras, how can I obtain the real position of furnitures given the depth images. Is there anyway I can find the camera parameters in the configuration files?

I also found the following code in demo_vision.py, for i, body in enumerate(env._object_names): pos = qpos[body][:3] It seems I can get the positions in world coordinate, how can I get the positions in different camera coordinates? And the number of pos output (10 pos ) seems larger the number of furnitures (8 parts). Thanks a bunch!

[edwhu]

Hello, The only information you have about the camera is from the Mujoco Camera element in the XML. I did a quick search and it seems like you have to compute the camera matrix yourself. https://github.com/openai/mujoco-py/issues/271

I also found a link on converting mujoco 3d coordinates to 2d camera here. https://github.com/yusukeurakami/mujoco_2d_projection

I would recommend trying this out without Unity rendering first by setting --unity False and see if the conversion works for the MuJoCo image. Then you can see if the Unity image also works. I suspect the Unity conversion might be slightly off because we hard code a FOV when we convert Mujoco camera to Unity camera.

Also that is weird, can you give me the exact script / parameters you used for demo_vision?

[gaopeng5]

Thanks for your help! I set the camera mode in floor_arena.xml, the corresponding camera id in furniture.py and collect depth image through demo_vision.py. I obtain the depth map with depth element in [0,1]. I would like to recover the real depth value in camera coordinate from the depth map.