search

ubi-agni / mujoco_ros_pkgs

Wrappers, tools and additional API's for using ROS with MuJoCo
68 stars 12 forks source link

Scene passed in renderingCallback is not initialized #17

Closed fpatzelt closed 1 year ago

fpatzelt commented 1 year ago

I noticed that the rendering of contact surfaces stopped working. The scene passed to the renderCallback has a geom buffer with size 0 (scene->maxgeom = 0). Either mjv_makeScene is never called or a different scene is passed to the renderCallback.

DavidPL1 commented 1 year ago

The problem was that the environment just used its own scene for the plugin render callbacks, which is only initialized when cameras are defined in the scene and offscreen rendering is turned on. The result would also only have been visible in the rendered camera images, not the simulation GUI. This is fixed in 7158fc0, i.e. the GUI rendering now passes the scene used to render the GUI frame.

I was wondering if we should include the option to let the plugins also render to offscreen scenes, because if you run the server headless the render callbacks currently don't get called. Example use case: recording a video with e.g. contact surface visualization on a headless setup for demonstration purposes.

On the other hand this could interfere with cameras used as data processing sources, so we should use a user parameter in the model xml to flag cameras that are only used for visualization and only run callbacks for them.

@rhaschke, what's your take on this?

rhaschke commented 1 year ago

I don't fully understand the problem yet. Let's talk tomorrow in person.