Rendered depth is upside down

[sriramsk1999]

Hello,

Thanks for the great work on RoboHive. I am observing a strange issue - the depth rendered by RoboHive is flipped upside down. The issue exists for all 4 cams (wrist, left, right, top) as well as other environments used directly through RoboHive or through torchRL's RoboHiveEnv.

Minimal example:

import gym
import robohive
import matplotlib.pyplot as plt
import numpy as np

env = gym.make("rpFrankaRobotiqData-v0")
env.reset()

extero_dict = env.get_exteroception()

rgb = extero_dict["rgb:left_cam:240x424:2d"]
dep = extero_dict["d:left_cam:240x424:2d"].squeeze()

plt.imsave("rgb.png", rgb)
plt.imsave("depth.png", dep)  ### Vertically flipped ###
plt.imsave("depth_flipped.png", np.flipud(dep))  ### Works ###

The corresponding images:

1. rgb.png
2. depth.png
3. depth_flipped.png

[vikashplus]

Glad you are enjoying working with RoboHive. To the best of my memory, we had to explicitly flip the buffer returned by the MuJoCo's python wrapper as it was upside down (see details here) It might be the case that the new wrapper doesn't require this explicit flipping.

It will be great if you can play around with it and send a PR if this is the case.

[sriramsk1999]

Great! Removing the line does the trick, I get aligned depth images now.

Also, a couple of issues with the segmentation rendering:

• It is also being flipped, I confirmed that this flip is not required either
• dpt_arr is being used in the segmentation condition, instead of seg_arr

I'd love to raise a PR, is there anything else to be checked? I'm a novice at this and not entirely how sure dm_control, mujoco and mujoco-py work together.

[vikashplus]

Once you have checked these details, please send a PR. CC @Jdvakil

[sriramsk1999]

Raised a PR to the dev branch. However, on the dev branch, the rendered depth is no longer in metres. It appears to have been scaled down, with the min/max values being 0.03500000387430191 and 0.03606634587049484. Not sure if this is expected behaviour.

[vikashplus]

Can you point me to the lines where you observe the mentioned scaling?

[sriramsk1999]

Fixed it :) The change introduced here setting the depth mapping method to mjDEPTH_ZERONEAR is causing the issue.

After removing this line, I get standard depth in metres. I am not getting any warnings either. What do you suggest?

[vikashplus]

Ah, yeah. I made that fix when Mujoco was a bit unstable on it. Can you send a PR for that? I'll test it again and accept the PR for the next release.

[sriramsk1999]

Gentle bump on this :) Please let me know if there's anything to be done from my side to get the PRs merged.

[vikashplus]

Thanks for the bump. All changes have been merged.