Deformable Visualization for Camera Sensors

[JShep-tri]

As it stands, there is no way to adequately visualize a deformable body in sim with a camera sensor.

There is currently an internal implementation of a DeformableCamera that I've written which visualizes (color and depth) deformable objects. The current constraints of this DeformableCamera is that each camera can only handle one deformable body. It also relies on some alterations to the render engine vtk API. Namely, the following functions (located within geometry/render_vtk/internal_render_engine_vtk.h) being moved from private to public, such that the DeformableCamera can call the DoRender... funcs and my custom RenderEngineVtkDeformable can call ImplementGeometry:

• void ImplementGeometry(vtkPolyDataAlgorithm* source, void* user_data);

• void DoRenderColorImage( const drake::geometry::render::ColorRenderCamera& camera, drake::systems::sensors::ImageRgba8U* color_image_out) const override;

• void DoRenderDepthImage( const drake::geometry::render::DepthRenderCamera& render_camera, drake::systems::sensors::ImageDepth32F* depth_image_out) const override;

• void DoRenderLabelImage( const drake::geometry::render::ColorRenderCamera& camera, drake::systems::sensors::ImageLabel16I* label_image_out) const override;

There are currently no alternatives to visualizing deformables - this is a pretty crucial feature for Drake to have given the increased support for deformables over the past year. Users will very likely wish to obtain sensor data from deformables.

There are certainly alternatives to the implementation I've described above that I feel need to be hashed out a bit more, but the DeformableCamera I've written is a good starting point.

[SeanCurtis-TRI]

All of this is a proxy to the true solution: teach RenderEngine about deformable geometry. Currently, it believes the world consists solely of rigid geometry.

Once that happens, "cameras" would take no particular role in considering deformable geometries (which is just and proper).

[JShep-tri]

All of this is a proxy to the true solution: teach RenderEngine about deformable geometry

Agreed, so I guess the question is, do we want to work in an intermittent solution (à la DeformableCamera, RenderEngineVtkDeformable) in the meantime, or focus on the "correct" solution which will take extra cycles?

[jwnimmer-tri]

Mostly this is a question for +@xuchenhan-tri on prioritization, I think?

[SeanCurtis-TRI]

Status update:

As of this posting we have partially achieved this feature. We can render deformable meshes. We can even use deformable meshes as a deformation basis for deforming embedded meshes (termed "driven meshes" in Drake). This functionality represents a big step from "not rendering at all" to "there's a narrow path".

We still need to broaden that path.

• [ ] Currently, only RenderEngineGl can do anything with deformable geometries. We need teach RenderEngineVtk about deformable meshes.
• [ ] Driven meshes currently just come from .obj file. Drake is trending towards preferring glTF files. We need to support glTF as driven meshes. (Current Drake lacks a mapping from glTF to RenderMesh, the data structure currently used as the in-memory mesh representation for defining driven meshes).
• [ ] examples/multibody/deformable/bubble_gripper.cc uses "package://drake_models/wsg_50_description/meshes/textured_bubble.obj". Once RenderEngines can use glTF files as driven meshes, swap this to texture_bubble.gltf and remove the .obj file in RobotLocomotion/models.