[geometry] Geometry owned *outside* of SceneGraph - MeshPainter system

[SeanCurtis-TRI]

Overview

There is a higher principle that is going to be initially explored using a particular application. The higher principle is this: there is some data that geometry queries depend on that SceneGraph doesn't own. Some examples of this principle:

• Deformable meshes: a system can encode the deformation of an object and produce a mesh. That mesh is provided dynamically to SceneGraph which can include it in proximity, perception, or illustration operations.
• Mesh painting: creating an image that represents contact between two geometries. One geometry is a painter object, one is a canvas object. The canvas object has a texture applied to it. All contacts between painter and canvas get "painted" onto the canvas's texture. (This is the system that is being used to exercise/explore the larger principle.)

MeshPainterSystem

The MeshPainterSystem behaves as described above. The system has been completely prototyped (such that a painted mesh can be observed in renderings (both color and label renderings). To see the results, bazel run examples/scene_graph:painting_a_mesh in the prototype branch (launch drake_visualizer first).

Implementation

The implementation of MeshPainterSystem is straighforward.

• Canvas and painter geometry objects get registered with SceneGraph (the owner can be any other system).
• MeshPainterSystem is constructed to know the ids of the canvas and painter geometries.
  • It uses SceneGraph to learn about the two geometries and creates two internal geometric representations of the canvas and painter geometries.
• It has a single input port (QueryObject from SceneGraph)
• It has a single output port (Image)
  • To calculate its output port
    • Evaluate the input port to find out the poses of the canvas and painter geometries.
    • Compute the contact patch in image space, and rasterize it into the image.

Integration

• Resolve algebraic loops
  • The MeshPainterSystem introduces an inadvertant algebraic loop. It depends on QueryObject to get poses. But the QueryObject depends on the painter system output texture in order to draw the texture. This is resolved by splitting QueryObject into domains of concern (discussed in #13241). The prototype branch has an implementation of this split, but not the implementation that will merge in master.
• Getting the image into SceneGraph
  • in the prototype branch, SceneGraph introduces a new kind of abstract-value input port. It consumes a new class InputImage (the type that MeshPainterSystem puts into its output port).
  • The user declares that they want to provide an input image (which creates the port) and then needs to connect to that port (otherwise a runtime error is thrown)
• Getting the image into RenderEngineVtk (the only RenderEngine that currently supports MeshPainterSystem)
  • The RenderEngine is the ultimate consumer of the InputImage (in this application).
  • Prior to performing relevant Perception rendering queries, InputImage ports get evaluated and the RenderEngine is given the chance to update its internal state to reflect the updated image.
  • RenderEngineVtk learns about input images based on special properties assigned to the geometry's PerceptionProperties. The semantics are alluded to here.

Open design questions

InputImage serial number

Swapping textures in the RenderEngine isn't cheap. We'd like to do it only if it's worthwhile. To that end, the InputImage has a serial number. Every time the owner modifies the image, it should "bump" the serial number. The RenderEngine can track the serial number of the input image it has in its own memory -- only if its stored serial number doesn't match the input serial number will it change its internal texture. For painting, this makes sense because if there's no contact, there's no change to the image. Only in the presence of contact between canvas and painter is there a chance for the image to change.

Is this premature optimization?

Nature of SceneGraph input ports

What should the semantics of the InputImage port be? In the prototype, it's a single port takes a single image. SceneGraph already has an external dependency on frame pose. However, it aggregates all of the frame poses from a single source in a single input port. Should this pattern be spread throughout all other types of geometric quantities? Images? Meshes? Alternatively, maybe we don't expect a single source to output an entire family of images? A vector of type seems to make the most sense -- a singleton vector is easy to create and leaves the door for easy aggregation.

Arbitrary image format support

Currently, the prototype requires the image provided at the port to be ImageRgba8U. In this specific context, it doesn't make a great deal of sense; a single-channel image would be enough to serve as a mask. However, this will require additional infrastructure in terms of declaration and type erasure so all images can come through but the expectations about what kind of image is provided can still be run through.

In the name of expedient prototyping, I didn't spend any time thinking about resolving this. It may be trivial or not. It seems to be the right thing to do, and it may prove that the prototype place holder might need to radically change.

Specify painting in URDF/SDF

Currently, there is no way to specify the necessary perception properties in URDF/SDF files. Again, probably straightforward, it just hasn't been tackled.

Expected PR train (loosely)

1. Modify QueryObject to eliminate the inevitable fallacious algebraic loop.
2. Add the initial painter shader to RenderEngineVtk such that we can use a static image to modify the geometry's appearance.
3. Extend SceneGraph to accept input images and feed them through to RenderEngine (extending RenderEngineVtk painter shader to use dynamic images (as opposed to just static).
4. Introduce MeshPainterSystem and example scenario.

[SeanCurtis-TRI]

cc @siyuanfeng-tri @DamrongGuoy @sherm1 @calderpg-tri

[DamrongGuoy]

I don't know enough to understand. Is it possible to have some pictures or diagrams to help the audiences?

[jwnimmer-tri]

Maybe we should close this issue as "not planned anytime soon"?