Closed SeanCurtis-TRI closed 5 months ago
cc @siyuanfeng-tri @DamrongGuoy @sherm1 @calderpg-tri
I don't know enough to understand. Is it possible to have some pictures or diagrams to help the audiences?
Maybe we should close this issue as "not planned anytime soon"?
Overview
There is a higher principle that is going to be initially explored using a particular application. The higher principle is this: there is some data that geometry queries depend on that
SceneGraph
doesn't own. Some examples of this principle:MeshPainterSystem
The
MeshPainterSystem
behaves as described above. The system has been completely prototyped (such that a painted mesh can be observed in renderings (both color and label renderings). To see the results,bazel run examples/scene_graph:painting_a_mesh
in the prototype branch (launchdrake_visualizer
first).Implementation
The implementation of
MeshPainterSystem
is straighforward.SceneGraph
(the owner can be any other system).MeshPainterSystem
is constructed to know the ids of the canvas and painter geometries.SceneGraph
to learn about the two geometries and creates two internal geometric representations of the canvas and painter geometries.QueryObject
fromSceneGraph
)Image
)Integration
MeshPainterSystem
introduces an inadvertant algebraic loop. It depends onQueryObject
to get poses. But theQueryObject
depends on the painter system output texture in order to draw the texture. This is resolved by splittingQueryObject
into domains of concern (discussed in #13241). The prototype branch has an implementation of this split, but not the implementation that will merge in master.SceneGraph
SceneGraph
introduces a new kind of abstract-value input port. It consumes a new classInputImage
(the type thatMeshPainterSystem
puts into its output port).RenderEngineVtk
(the onlyRenderEngine
that currently supportsMeshPainterSystem
)RenderEngine
is the ultimate consumer of theInputImage
(in this application).InputImage
ports get evaluated and theRenderEngine
is given the chance to update its internal state to reflect the updated image.RenderEngineVtk
learns about input images based on special properties assigned to the geometry'sPerceptionProperties
. The semantics are alluded to here.Open design questions
InputImage
serial numberSwapping textures in the
RenderEngine
isn't cheap. We'd like to do it only if it's worthwhile. To that end, theInputImage
has a serial number. Every time the owner modifies the image, it should "bump" the serial number. TheRenderEngine
can track the serial number of the input image it has in its own memory -- only if its stored serial number doesn't match the input serial number will it change its internal texture. For painting, this makes sense because if there's no contact, there's no change to the image. Only in the presence of contact between canvas and painter is there a chance for the image to change.Is this premature optimization?
Nature of SceneGraph input ports
What should the semantics of the
InputImage
port be? In the prototype, it's a single port takes a single image.SceneGraph
already has an external dependency on frame pose. However, it aggregates all of the frame poses from a single source in a single input port. Should this pattern be spread throughout all other types of geometric quantities? Images? Meshes? Alternatively, maybe we don't expect a single source to output an entire family of images? A vector of type seems to make the most sense -- a singleton vector is easy to create and leaves the door for easy aggregation.Arbitrary image format support
Currently, the prototype requires the image provided at the port to be
ImageRgba8U
. In this specific context, it doesn't make a great deal of sense; a single-channel image would be enough to serve as a mask. However, this will require additional infrastructure in terms of declaration and type erasure so all images can come through but the expectations about what kind of image is provided can still be run through.In the name of expedient prototyping, I didn't spend any time thinking about resolving this. It may be trivial or not. It seems to be the right thing to do, and it may prove that the prototype place holder might need to radically change.
Specify painting in URDF/SDF
Currently, there is no way to specify the necessary perception properties in URDF/SDF files. Again, probably straightforward, it just hasn't been tackled.
Expected PR train (loosely)
QueryObject
to eliminate the inevitable fallacious algebraic loop.RenderEngineVtk
such that we can use a static image to modify the geometry's appearance.SceneGraph
to accept input images and feed them through toRenderEngine
(extendingRenderEngineVtk
painter shader to usedynamic
images (as opposed to just static).MeshPainterSystem
and example scenario.