search

RobotLocomotion / drake

Model-based design and verification for robotics.
https://drake.mit.edu
Other
3.26k stars 1.26k forks source link

[MuJoCo Parser] Support for applying texture (for example from images) directly to primitive geometries #21958

Open agarwal-abhinav opened 1 day ago

agarwal-abhinav commented 1 day ago

Is your feature request related to a problem? Please describe. Currently, the MuJoCo parser doesn't support applying texture directly to primitives. For instance, if a geometry is specified by primitives in the .xml file, and a texture is added to it via (say) a .png specified in the .xml, drake doesn't currently support loading that. This requires reasoning about converting the primitives to .obj, the texture mapping to .mtl, and then modifying the .xml to load the .obj. This is tedious pre-processing and doing it without human in the loop requires reasoning about u-v coordinate maps in a generic way, which isn't always convenient given multiple options for mapping texture in the .xml.

Describe the solution you'd like Currently, drake supports adding texture directly to primitive geometry in .sdf files for visualization in camera rendering.

  1. First suggestion is to add the same feature for parsing mujoco .xml.
  2. Second suggestion is to add the feature to visualize it meshcat as well. Doing this will not only allow rendering of the scene from camera, but also visualization of the robot trajectories.

Describe alternatives you've considered Even if the mapping to .obj works well for visualization in meshcat, it doesn't always work well in camera rendering (or vice-versa). Attached is an image as an example. Additionally, the right wall in the meshcat rendering can make it clear that getting u-v coordinates right isn't the most effortless pre-processing.

Additional context There is some elementary discussion about this on the drake developers slack. cc @SeanCurtis-TRI The attached example is part of an effort to load robocasa environments in drake for combining learning based robotics (needing a large library of feature rich yet varied environments) with model based tools like GCS. cc @GenericP3rson who has been working on this and provided the attached visualization. This issue is closely related to the work being done in #20444 cc @RussTedrake

IMG_8151

SeanCurtis-TRI commented 1 day ago

This is related to #19076. When extracting material definitions from the mjcf file, they'll inevitably end up in GeometryProperties. However, those don't make it to Meshcat (as per the linked issue). So, the solution for this issue requires resolution of both issues.