RobotLocomotion / drake

Model-based design and verification for robotics.
https://drake.mit.edu
Other
3.28k stars 1.26k forks source link

Meshcat support for arbitrary object_json #20941

Open jwnimmer-tri opened 7 months ago

jwnimmer-tri commented 7 months ago

Is your feature request related to a problem? Please describe.

Meshcat uses three.js under the hood (see https://threejs.org/docs/). In that ecosystem, there are a huge number of very good special-purpose geometries and objects and materials defined beyond anything we'd ever introduce as a shape description in SceneGraph. So far, we've solved that by adding special-purpose SetObject overloads (or pseudo-overloads) -- e.g. Meshcat::SetLine to add a linestrip.

Functions like that are fine, but it means that anytime a user wants to access a not-yet-wrapped feature of three.js objects, they/we need to add a new Meshcat C++ function and C++ msgpack definition and bind it into pydrake and ... etc.

As an alternative, we should offer an "escape hatch" for advanced users who want access to everything three.js has to offer, where they just hand us the JSON object they want and we pass it along, unchanged.

To be clear -- we would still aim to cover the common use cases (e.g., linestrips) with sugared-up functions that make the simple things easy. The goal here is to make the advance things at least possible.

Describe the solution you'd like

Something like:

 /** Sets the 3D object at a given `path` in the scene tree to the given JSON,
  per the https://threejs.org/docs/#api/en/loaders/ObjectLoader threejs format. */
  void SetObjectRaw(std::string_view path, std::string_view threejs_object_json);

Describe alternatives you've considered

The only one I can think of is the existing condition: add bespoke functions one by one, for each kind of Object3D a user wants.

SeanCurtis-TRI commented 6 months ago

Just to pin a relevant thought here:

One of the things sugar does is handle the difference between Drake's z-up world and three.js's y-up world. When we add this feature, we'll want to make sure that responsibility abundantly clear in its documentation.