RobotLocomotion / drake

Model-based design and verification for robotics.
https://drake.mit.edu
Other
3.26k stars 1.26k forks source link

Meldis ignores GeometryProperties (phong, diffuse_map) #19077

Open jwnimmer-tri opened 1 year ago

jwnimmer-tri commented 1 year ago

See #18844 for overall context.

When GeometryProperties contains (phong, diffuse_map), it's a filename for a texture map (e.g., png file). We should transmit that property to Meldis somehow, and have it be used in the display. Currently, we just drop the information on the floor.

Note that Meldis does obey the OBJ file's associated MTL file which can contain a map_Kd texture. This issue is only relevant to OBJs without an MTL.

jwnimmer-tri commented 1 year ago

I have a couple of ideas how to pass extra data to Meldis.

In the extreme, I just add a new message type. The contract we need for Meldis is that old logs still play back correctly on current visualizers (so, we can't change the schema of any current lcmtypes), but not that new logs play back on old visualizers. So, we could send an additional GEOMETRY_PROPERTIES message or something which Meldis can listen for.

Alternatively (and my current plan), we can overload "string_data" to be either a bare filename (as currently for meshes) xor a json string (dictionary) and have Meldis treat the two separately. If the case of a dictionary, one key would be filename (when using a mesh), but then we can have arbitrary keys beyond that (like geometry_properties: { ... }).

SeanCurtis-TRI commented 1 year ago

The string_data approach might break backwards compatibility. While it's been a while since I looked, when I added hydroelastic representations to DrakeVisualizer, I added this comment. If it's correct, a non-empty string_data will be interpreted by drake_visualizer as a file path.

jwnimmer-tri commented 1 year ago

Yup. I'd patch drake_visualizer to snatch the filename. We only need to interop with contemporary versions of drake_visualizer (i.e., the one we ship with any given Drake).

jwnimmer-tri commented 5 months ago

And now that the director-based drake_visualizer is gone, we can go to JSON right away. The convention could be that if string_data's first character is { then we parse it as JSON, otherwise the historical convention applies (e.g. mesh filename).

SeanCurtis-TRI commented 1 day ago

Note: we've opened the door for json-valued messages. #21952 uses it to transmit in-memory meshes across the wire so that meldis can load them into its own Meshcat.