mujoco rendering offscreen get image array with all the same value

[ZiwenZhuang]

Describe the bug Certain behavior calling env.sim's state function will cause the later rendering image array with only the same value. The behavior is described in the script.

To Reproduce

import numpy as np
import matplotlib.pyplot as plt
import mujoco_py
import gym

env = gym.make("FetchReach-v1")
env.env.viewer == None
# For the confirm that 'env.env.viewer' is 'None'

env.env.viewer = mujoco_py.MjViewer(env.env.sim)
# Without the step above, there will be a "GLEW initialization error". Issue has been submitted.
env.env.sim.render(width=48, height=48, mode='offscreen')
env.env.reset()

env.env._get_viewer().render()

img = env.env.sim.render(width=48, height=48, depth=False)
plt.imshow(img)
plt.show()
# The image showed here is normal
env.env._get_viewer().render()

# Then start to set the state from given mujoco interface
nq = env.env.sim.model.nq
nv = env.env.sim.model.nv

qpos = gym.spaces.Box(high=np.ones(nq), low=-np.ones(nq), dtype=np.float32).sample()
qvel = gym.spaces.Box(high=np.ones(nv), low=-np.ones(nv), dtype=np.float32).sample()

old_state = env.env.sim.get_state()
new_state = mujoco_py.MjSimState(old_state.time, qpos, qvel, old_state.act, old_state.udd_state)
env.env.sim.set_state(new_state)

env.env.sim.forward()
# finish set_state

env.env.sim.render_contexts[0]._set_mujoco_buffers()

img = env.env.sim.render(width=48, height=48, depth=False)
plt.imshow(img)
plt.show()
# Then the image showing has only one color

Expected behavior The last image plotted should be a meaningful image.

Error Messages No error message occurred

Desktop (please complete the following information):

• OS: Ubuntu 18.04
• Python Version: Python 3.5.2 |Continuum Analytics, Inc.| (default, Jul 2 2016, 17:53:06)
• Mujoco Version: [1.50]
• mujoco-py version: '1.50.1.68'

Environment echo $LD_LIBRARY_PATH /opt/ros/melodic/lib:/home/leo/.mujoco/mjpro150/bin/:/home/leo/.mujoco/mujoco200/bin/:/home/leo/.mujoco/mujoco200_linux/bin/:/usr/local/cuda/lib64/:/usr/local/cuda-10.0/lib64/

Additional context None

[ZiwenZhuang]

Partially solved, By removing the code env.env.sim.render_contexts[0]._set_mujoco_buffers(), it seems work. But due to the OpenGL issue (from issue 408 and issue 432, even you provide it an on-screen viewer before any offscreen viewer, the rendering could still generate all-black image on both numpy arrays (offscreen render) and mujoco-py window in some frames.

[ZiwenZhuang]

Solved: referring to https://github.com/openai/mujoco-py/issues/408 and https://github.com/openai/mujoco-py/issues/383

[bibbygoodwin]

Just wanted to add a note to this in case it helps anyone coming across the issue.

I was having a similar problem in a setting where I had 2 sims with offscreen render contexts:

sim1 = mujoco_py.MjSim(panda_model, nsubsteps=3)
viewer1 = mujoco_py.MjViewer(sim1)
sim2 = mujoco_py.MjSim(sawyer_model, nsubsteps=3)
viewer2 = mujoco_py.MjViewer(sim2)

render_context1 = mujoco_py.MjRenderContextOffscreen(sim1)
sim1.add_render_context(render_context1)
render_context2 = mujoco_py.MjRenderContextOffscreen(sim2)
sim2.add_render_context(render_context2)

I would step and render from these sims sequentially (sim1, then sim2, then sim1 again and so on). However, a call to the sim1 render (with data = sim1.render(720, 720, camera_name='sideview', depth=False)[::-1, :, :]) would result in a broken image - either of the sim2 model from a weird angle, or sometimes I believe of the sim1 model but again from a weird angle. Calling sim1.render_contexts[n]._set_mujoco_buffers() with n=1 or n=2 (unsure of what these two different render contexts are - they both seem to refer to the offscreen context) fixed things.